Upgrading to Oracle 11gR2
-
Upload
syed-hussain -
Category
Technology
-
view
3.990 -
download
1
description
Transcript of Upgrading to Oracle 11gR2
Do I actually need to care about upgrading my existing environment to race with
Oracle latest release
Is it mandatory
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 1
Does it really matter which version of Oracle do we use
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 2
Upgrading to Oracle 11gR2
A Successful case study
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3
We really had a roller coaster journeyhellip
A successful upgrade
A successful downgrade
De-installed failed GI home before start over
Manual upgrade
Tried out of the box optionshellip
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4
Disclaimer
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5
This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied
This material should not be reproduced or used without the authors written permission
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Does it really matter which version of Oracle do we use
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 2
Upgrading to Oracle 11gR2
A Successful case study
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3
We really had a roller coaster journeyhellip
A successful upgrade
A successful downgrade
De-installed failed GI home before start over
Manual upgrade
Tried out of the box optionshellip
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4
Disclaimer
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5
This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied
This material should not be reproduced or used without the authors written permission
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrading to Oracle 11gR2
A Successful case study
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3
We really had a roller coaster journeyhellip
A successful upgrade
A successful downgrade
De-installed failed GI home before start over
Manual upgrade
Tried out of the box optionshellip
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4
Disclaimer
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5
This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied
This material should not be reproduced or used without the authors written permission
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
We really had a roller coaster journeyhellip
A successful upgrade
A successful downgrade
De-installed failed GI home before start over
Manual upgrade
Tried out of the box optionshellip
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4
Disclaimer
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5
This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied
This material should not be reproduced or used without the authors written permission
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Disclaimer
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5
This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied
This material should not be reproduced or used without the authors written permission
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Foreword
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6
The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Know your presenter
Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters
EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Know your presenter
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9
Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Why upgrade ndash A burning question
Do we really need to care which version are we using
Is it mandatory to upgrade an existing environment with every Oracle new release
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Why upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13
To best of my knowledge the following situations may contributes to oppose an upgrade
Legacy application Doesnrsquot want to take the risk Lack of skills and support
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Why upgrade
Situations that are likely to invite you to embrace for an upgrade
Organization standards Policy standards by Oracle for the
old releases Tantalizing new features Bug fixes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade compatibility matrix
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18
Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2
Source Database Target Database9208 or higher 112x
10105 or higher 112x
10202 or higher 112x
11106 or higher 112x
The following database version will require an indirect upgrade path
Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x
805 (or lower)----gt 806 ---gt 9208 ----gt112x
817 (or lower)----gt 8174---gt 10204----gt112x
9013 (or lower)----gt 9014-- -gt10204----gt112x
9207(or lower)----gt9208----gt112x
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade pattern [BEST PRACTICE]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20
UPGRADEClusterware
+ ASM
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade pattern [best practice]
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23
UPGRADEClusterware
+ ASM
INSTALLRDBMS
Software
UPGRADEdatabases
Post upgrade
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25
HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4
CISCO Switch
NetBackup 656
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27
8 nodes
Production 1 Production 2
6 nodes
7 nodes
Non-Production 1 Non-Production 2
3 nodes
bull31 databasesbull51 instances
bull4 logical processorsbull22GB Memory
bull11 databasesbull20 instances
bull3 logical processorsbull12GB Memory
bull60 databasesbull80 instances
bull3 logical processorsbull16GB Memory
bull12 databasesbull20 instances
bull3 logical processorsbull12GB Memory
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
The framework that we have followed for the upgrade
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview about our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupdate
The framework that we have followed for the upgrade
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Overview of our environment
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30
V1123varopt
V1131varopt
HP UX upgrade from V1123 TO v1131
EMC Storage Configurationupgrade
ClusterwareDatabasesupgrade
The framework that we have followed for the upgrade
10204 11202
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32
Preparation is the headstone to every successful installation orupgrade projects
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33
Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location
Must set GI home different from OB Configure Single Client Access Name(SCAN)
o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file
root user access UNSET the following
o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34
Spaceo Additional space for Oracle software
home typically u00o Enough tmp space
Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR
CRS versiono Remember the crs active and software version
must be the samecrsctl query crs activeversion
crsctl query crs softwareversion
Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list
ndashfixup ndashfixupdir ltdirnamegt -verbose
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Prepare for upgrade ndash prerequisites checks
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35
Verify nodes readiness for upgrade
While upgrading from pre 11gR2 use the following syntax
runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose
While upgrading 11201 to 11202 or higher use the following syntax
runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling
-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME
-dest_version 112030
ndashfixup ndashfixupdirpath ltdirnamegt -verbose
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware ASM upgrade ndash Whatrsquos new
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37
Grid Infrastructure
Out-of-Place andIn-place upgrades
New GI home
Clusterware
Automatic Storage Management (ASM)
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware ASM upgrade
Is it necessary to upgrade Clusterware AND ASM together
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39
Clusteware +
ASM = no rolling upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51
ORACLE BASE must be different from the SW
locationhelliphellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54
Cause False warning generated by unpublished CVU bug 10024549
Solution The warning can be ignored
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61
Can I run the rootupgradesh script in PARALLEL across nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62
Can I run the rootupgradesh script in PARALLEL across nodes
The answer is YES howeverhellip
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63
Always run the script on the LOCAL node first
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65
Always run the script on the LOCAL node first
Run the script in parallel on all nodes except the LAST node
Run the script on the LAST node
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66
Shuts down the existing clusterware stack
Brings up the new version clusterware stack
Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version
rootupgradesh
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68
Average rootupgradesh execution time on each node was just about 15 min
Took more time on couple nodes mostly while stopping the existing cluster stack
Look closely for the logs under u00app1120gridcfgtoollogs
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69
rootupgradesh
execution failed
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]
How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community
References
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71
Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade
u01app1120gridrootupgrade ndashforce (gt=11202 only)
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76
Internet
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77
Internet MOS
Cried for HELP
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78
ASM alertlog trace files amp log files
Internet MOS
Cried for HELP
Doesnrsquot help much
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain Slide 79
Life Savior
MOS [ID 3983741]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80
SQLgt ORACLE instance shut downSQLgt Disconnected
[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8
2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009
Copyright (c) 1982 2007 Oracle All Rights Reserved
Enter user-name Connected to an idle instance
SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected
cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81
What might have gone wrong
ASM instance name registered as ASMninstead of +ASMn in the OCR
Could be due to a manual entry
GI is very particular about + in contrast toprevious cluster version
Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82
Remedy
Stop ASM and dependent instances
Remove dependency on ASM instance
Remove ASM from OCR
Add ASM back in with lsquo+rsquo correct name
Add dependency to ASM
Startup ASM
Resume the failed step
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84
In action
srvctl stop asm ndashn lthostnamegt
srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f
[do it for all active instances on the node]
srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt
srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt
-o ltORA_ASM_HOMEgt
srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt
-s +ASM_INSTANCE1
[do it for all active instances on the node]
srvctl start asm ndashn lthostnamegt
Repeat on all nodes
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87
Post upgrade checks ndash activesoftware version
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [112020]
crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [112020]
Ensure activerelease version reflect to 11202
for a successful upgrade
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88
Post upgrade checks ndash cluster stats across nodes
crsctl check cluster -all
node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89
Post upgrade checks ndash query voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows
Version 3
Total space (kbytes) 1051708
Used space (kbytes) 27048
Available space (kbytes) 1024660
ID 1837247566
DeviceFile Name devrdskoracleocrln1ora_ocr_001
DeviceFile integrity check succeeded
DeviceFile Name devrdskoracleocrln1ora_ocr_002
DeviceFile integrity check succeeded
DeviceFile not configured
DeviceFile not configured
DeviceFile not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
Up to FIVEOCR files
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90
Post upgrade checks ndash query voting disk
crsctl query css votedisk
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []
2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []
3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91
Post upgrade checks ndash bg and daemon processes
ps -ef |grep dbin
oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin
oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin
root 14539 1 0 151428 024 u00app1120gridbincssdagent
oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f
root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin
root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot
oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin
oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin
oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin
oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin
root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin
root 14595 1 0 151450 306 u00app1120gridbinoctssdbin
root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot
root 14533 1 0 151424 026 u00app1120gridbincssdmonitor
oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin
oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin
ps -ef |grep initd
root 66 0 0 Feb 9 000 pagetable_init_daemon
oracle 8645 15234 0 193102 pts0 000 grep initd
root 13977 1 0 151319 000 binsh sbininitdinitohasd run
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Clusterware and ASM upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92
Remember
You can easily resumedowngrade from a failed or successful upgrade
Can fall back to the pre 11gR2 version
You hardly need a 3-step procedure to do that
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Remove OLD homes
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93
Removing OLD cluster home
Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES
export ORACLE_HOME=$OLD_HOME
detach OLD_HOME
$OLD_HOMEouibinrunInstaller -detachHome
-silent -local ORACLE_HOME=$OLD_HOME
confirm $OLD_HOME is removed from central inventory
$OLD_HOMEOPatchopatch lsinventory -all
remove files in OLD_HOME manually
binrm -rf $OLD_HOME
unset ORACLE_HOME
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95
cluvfy stage -pre dbinst -n
node01node02node03 -d
$ORACLE_HOME ndashverbose
Verify the READINESS of the nodes for DB software installation
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112
How to manage the pre 11gR2 databases under 11gR2 GIH
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
11gR2 RDBMS software installation
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113
rootnode01 u00app1120gridbin
crsctl pin css -n node01 node02 node03
CRS-4664 Node node01 successfully pinned
CRS-4664 Node node02 successfully pinned
CRS-4664 Node node03 successfully pinned
PINNING NODES
You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade ndash Caution
Ensure a valid database backup BEFORE you kick off the db upgrade procedure
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade ndash upgrade paths
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116
UPGRADE
SQLgt catupgradesql
bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql
Tablespaces [make adjustments in the current environment]
--gt SYSTEM tablespace is adequate for the upgrade
minimum required size 788 MB
--gt UNDOTBS1 tablespace is adequate for the upgrade
minimum required size 43 MB
--gt SYSAUX tablespace is adequate for the upgrade
minimum required size 566 MB
--gt TEMP tablespace is adequate for the upgrade
minimum required size 61 MB
PURGE DBA_RECYCLEBIN
must be executed immediately prior to executing your upgrade
Recommendations
Oracle recommends gathering dictionary statistics prior to
upgrading the database
To gather dictionary statistics execute the following command
while connected as SYSDBA
EXECUTE dbms_statsgather_dictionary_stats
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
As a best practice run OH_11Grdbmsadminutlu112isql
Components [The following database components will be upgraded or installed]
--gt Oracle Catalog Views [upgrade] VALID
--gt Oracle Packages and Types [upgrade] VALID
--gt JServer JAVA Virtual Machine [upgrade] VALID
--gt Oracle XDK for Java [upgrade] VALID
--gt Real Application Clusters [upgrade] VALID
--gt Oracle Workspace Manager [upgrade] VALID
--gt OLAP Analytic Workspace [upgrade] VALID
--gt OLAP Catalog [upgrade] VALID
--gt EM Repository [upgrade] VALID
--gt Oracle Text [upgrade] VALID
--gt Oracle XML Database [upgrade] VALID
--gt Oracle Java Packages [upgrade] VALID
--gt Oracle interMedia [upgrade] VALID
--gt Spatial [upgrade] VALID
--gt Data Mining [upgrade] VALID
--gt Expression Filter [upgrade] VALID
--gt Rule Manager [upgrade] VALID
--gt Oracle OLAP API [upgrade] VALID
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123
Listener will be created
on all nodes
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126
Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS
$ cd $ORACLE_HOMErdbmslib
$ make -f ins_rdbmsmk dv_off
$ cd $ORACLE_HOMEbin
$ relink all
(For RAC installations run these commands on all nodes)
cd u00apporacleproduct1120db_1rdbmslib
make -f ins_rdbmsmk dv_off usrccsbinar d
u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo
usrccsbinar cr
u00apporacleproduct1120db_1rdbmsliblibknlopta
u00apporacleproduct1120db_1rdbmslibkzvndvo
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database post upgrade ndash list
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135
Run utlrpsql to compile ANY invalid objects
Run utlu112ssql to check post upgrade components status
Adjust time zone data in the database
Gather statistics for system and fixed objects
Amend scripts [if any] to affect the new OH settings
Verify the database upgradesrvctl config database ndashd ltDBNAMEgt
Back up the database
Modify the COMPATIBLE init Parameter to 11gR2
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade ndash measuring upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade ndash trivia
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137
Upgraded over 100 databases successfully using MANUAL and DBUA methods
Avg database upgrade time was just about 2 hrs
Upgrade using DBUA for RAC DB proved a wise decision
Nearly identical duration between CLI and DBUAupgrade methods
Duration is typically proportional to the number ofinstalled database components
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Database upgrade ndash CLI upgrade steps
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138
Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Upgrade aftermath stories
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140
Snapshot Controlfile must be placed on a shared storage for a RAC database
ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used
grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL
ADD_PRIVILEGEASSIGN_ACL
ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]
IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists
ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
Very important apply the latest PSU patch right after
your upgrade
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
What will be covered
o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
References
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143
Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]
Oracle 11g R1R2 Real Application Clusters Essentials
Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom
A big thank to red-gate
and you all
for listening
Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144
You can write me at sjaffarhussaingmailcom