Powerful New Ways to Use Oracle Data Guard for Planned … · Author: Larry Carpenter Created Date:...

1 Copyright © 2011, Oracle and/or its affiliates. All rights

reserved.

Powerful New Ways to Use Oracle Data Guard for

Planned Maintenance Larry M. Carpenter Distinguished Product Manager

Oracle


reserved.

Program Agenda

• Things to think about first

• Planned Maintenance other than the database

• Standby First Patching for the Oracle Database

• Rolling Upgrades for Oracle Database Versions

• When SQL Apply cannot be used

• All the way to Zero Downtime Upgrades or Migrations


reserved.

Downtime

• We all want zero downtime upgrades and migrations

• Planned Maintenance downtime for the database will

vary based on the method you use

– Your total downtime will depend on your configuration

Downtime Downtime Downtime


reserved.

Complexity

• The lower the complexity the better, of course

– Higher complexity means more potential for error

• But as with everything in the HA/DR environment,

Complexity is directly connected to Downtime

– The Simpler the mechanism the higher the Downtime

Complexity Complexity Complexity


reserved.

Risk

• Risk, should be the first consideration

– But everybody thinks about Downtime first

• There is also a connection between Risk and Complexity

– Simpler mechanisms equate to lower Risk

• Fewer moving parts

• Less room for error

Risk Risk Risk


reserved.

Considerations

• What everybody wants:

– Downtime = Zero

– Complexity = Simple

– Risk = Low

• What the reality is:

– You can‟t get Zero Downtime without increased Complexity and Risk

In Summary


reserved.

Program Agenda








reserved.

Planned Maintenance Events

• Hardware Firmware Patches

– System, Storage, Network

• Operating System Patches and Upgrades

• Oracle Software • Oracle Clusterware Patching and Upgrading

• ASM Patching and Upgrades

– These software patches may be applied in a rolling fashion

• As of Oracle Database 11g Release 2 Clusterware and ASM are in the

same Grid Infrastructure home which allows for out of place upgrades


reserved.

Rolling RAC/ASM Patches

• For patches check the Patch README first

– “This patch is RAC Rolling Installable”

• RAC rolling installable – Per node

Stop Instances

Stop CRS

Start CRS

Patch Start

Instances

Downtime Complexity

The Risk is elevated because you are

modifying your production system Risk


reserved.

Mitigating the Risk for Rolling Patches

• 100% Replica of the Entire Production system

• Exclusive time to perform full load validation.

• Production data set with identical schema statistics.

– Real world production transactions and client concurrency.

• Production/Test System AWR data for comparison

• Operational and availability test suites.

Testing Requirements

Downtime Complexity

The Risk goes down as long as you

can meet these testing requirements Risk


reserved.

Data Guard & Planned Maintenance Essential for High Availability

LAN & MAN deployments provide Local HA and DR

Extend to a Wide Area Network and add remote DR


reserved.

Reducing Planned Downtime with Data Guard

• Data Guard primary with standby database

• Defer redo shipping

• Implement changes at standby system

• Data Guard resynchronizes standby with primary

• Validate changes

• Switchover (Typical duration 60-120 seconds)

• Production on new environment

High Level Overview


reserved.

Technology Refresh and Migrations

• Data center moves

• Technology refresh or Changes

– 32bit 64bit or Windows Linux, etc.

• See Support Note 413484.1 for supported configurations

– Single node Oracle RAC

– Migrating to ASM

• Upgrading RAC or ASM

• Operating system and/or hardware maintenance

Using Redo Apply – Physical Standby


reserved.

Lower the Risk while keeping Downtime Low

• Changes on Standby Systems

• Test Changes

• Switchover to standby

Application Tier

Data Guard

Switchover


reserved.

With a Local ‘HA’ Standby

Application Tier

Post Switchover

Data Guard

• Production Running on

updated systems

• Effect changes on original

system and switchover

Complexity Risk Downtime


reserved.

If You Do Not Have a Local ‘HA’ Standby

Application Tier

Data Guard

Switchover


reserved.

You have to Restart the Middle Tier

Application Tier

Post Switchover

Data Guard

Complexity Risk

The Downtime is higher due to

Middle Tier restart Downtime


reserved.

Rolling RAC/ASM Patches

• RAC rolling installable – Per node

• When your testing period is done, Switchover!

• This also applies to full software upgrades!

Performed on the Standby System First!

Stop Instances

Stop CRS

Start CRS

Patch Start

Instances

Complexity Risk

The Risk is low at the cost of a little

more downtime for the switchover Downtime


reserved.

Program Agenda








reserved.

Standby First Patching

• Benefit

– Apply patches to standby and test before applying to primary

• All 11.2.0.1 BP7 and later: • Patch Set Exception (PSE)

• Critical Patch Update (CPU)

• Patch Set Update (PSU)

• Oracle Exadata Database Machine bundled patch

• Oracle Exadata Storage Server Software patch

• See MOS Note 1265700.1


reserved.

Patch the Standby ‘First’

• Patch the Standby

• Test Changes!!!!

• Switchover (or not)

Application Tier

Data Guard


reserved.

Without any Patch SQL to Run

Application Tier

Post Switchover

Data Guard

• Production up after Switchover

Complexity Risk

• Patch original Primary

• Switchover again (or not)

Downtime


reserved.

With Patch SQL to Run

Application Tier

Post Switchover

Data Guard


• Run the Patch SQL first

• Then production is Ready

• Patch original Primary

• Switchover again (or not)

More Downtime due to the SQL


reserved.

Without Using a Switchover

• After testing the patch on the

standby „First‟ with Snapshot

Standby

• Patch the Primary and run the

SQL there

Application Tier

Data Guard

Complexity Risk Downtime More Downtime due to the SQL


reserved.

Program Agenda





• When SQL Apply Won't Work



reserved.

Database Rolling Upgrades

• Patches and PSUs that are not Standby First

• Complete Patch-sets and new database releases

• Two Methods

– Using SQL Apply

• Any upgrade from 10.1.0.3 onward

– Using Redo Apply

• Any upgrade from 11.1.0.7 onward

• Added convenience for physical standby users

– No need to create new logical standby just for upgrade


reserved.

Patchset Apply Downtime Comparison

© 2008 Oracle

27

SQL Apply Rolling Upgrade

1 minute +

Conventional Upgrade

~ 1 hour

Data Guard Switchover time

• 60-120 Seconds

• catupgrd.sql and utlrp.sql are still run

• But with no downtime

Catupgrd.sql

• 13-56 minutes

• Depends on CPU, I/O, DB objects

utlrp.sql

• Average: 3-6 minutes

• Compiles invalid objects

• Timings depend on # of invalid objects

Plus seconds, when no application restart and

using client failover best practices

Plus seconds or minutes to restart the middle

tier if you had to stop it to take down the

Production database


reserved.

Using SQL Apply

PROD SQL

APPLY

synchronize Install new Oracle version in

separate homes on A & B

LSTBY

synchronize SQL

APPLY PROD

Upgrade B, synchronize with

production, test to validate upgrade

SWITCHOVER

PROD SQL

APPLY

Switch production to B

zero downtime for read-only users

SQL

APPLY

synchronize PROD

Upgrade A and synchronize with

production

Database A Database B

release n release n+1


reserved.

Using Redo Apply

PROD REDO

APPLY

synchronize

Install new Oracle version in separate

homes on A & B, set guaranteed

restore point (GRP) on A

synchronize SQL

APPLY PROD

Temporarily convert B to use SQL

Apply, upgrade and sync

Database A Database B

SWITCHOVER

Switchover, flashback A to GRP,

mount in new/upgraded home PROD REDO

APPLY

synchronize PROD

Upgrade A via redo stream

and synchronize REDO

APPLY release n release n+1


reserved.

Database Rolling Upgrade Process

• Conceptually simple, but..

– For many DBAs, this is their first encounter with a logical standby

• Setting of parameters for performance

• Iterative testing

– Several parameters are specific for rolling upgrade

• record_unsupported_operations

• log_auto_delete must be FALSE etc..

– Publishing services following the switchover

– Rolling upgrade is a process, not a DDL


reserved.


• Shell script available via MetaLink Note 949322.1

• Incorporates best practices for the entire process

• Features

– Automatic migration of OCI clients using server-side TAF

– Can restart script after failure

– Progress checks

– Ability to abandon rolling upgrade mid-way

– Customizable (by definition)

– Repeatable testing (by definition)

Automated Workflow


reserved.


• Phase 1: Prepare – Routine checks and conversion of the standby

• Phase 2: Upgrade Standby and Synchronize – User upgrades the logical standby, Data Guard Synchronizes

• Phase 3: Switchover – Switchover to the transient logical standby

• Phase 4: Upgrade Original Primary – Flashback original primary so it becomes a physical

– Synchronize with the primary and get upgraded automatically

Four Phases


reserved.

Phase 1: Prepare

Phase 1: Convert standby to logical

Continue? [y/n]: y

Flashback enabled on both databases.

Converting BOSTON to a logical standby

Checking status of managed recovery process (MRP0) on BOSTON

MRP0 is running on BOSTON

Stopping MRP0 on BOSTON ...

MRP0 successfully stopped on BOSTON

Taking Guaranteed Restore Points on CHICAGO and BOSTON ...

ru_primary_before_upgrade (GRP) taken on CHICAGO

ru_stdby_before_upgrade (GRP) taken on BOSTON


reserved.

Phase 1: Prepare - continued

Taking data dictionary snapshot in the redo stream on CHICAGO ...

Dictionary snapshot successfully logged in the redo stream

Thread 1, sequence 7 contains the data dictionary snapshot

Converting BOSTON to a logical standby ...

Issuing alter database recover to logical standby keep identity ...

BOSTON successfully converted to a logical standby

Setting SQL Apply parameters ...

MAX_SGA set to 500M

MAX_SERVERS set to 50

RECORD_UNSUPPORTED_OPERATIONS set to TRUE

LOG_AUTO_DELETE set to FALSE

Starting SQL Apply on BOSTON ...

SQL Apply successfully started

Waiting for dictionary load to complete on BOSTON ...

Dictionary load progress 40% done

Dictionary load completed

BOSTON is currently 5 minutes behind CHICAGO

Estimated catch-up time 1 minute

Phase 1 of rolling upgrade successfully completed


reserved.

Phase 2: Upgrade Standby and Synchronize

Phase 2: Ready to upgrade the logical standby

Continue? [y/n]: y

Stopping SQL Apply in BOSTON ...

SQL Apply successfully stopped.

Please upgrade BOSTON using Upgrade instructions provided in

Oracle Upgrade Guide of the target version. Once you have upgraded

BOSTON, please say “y” to continue.

Continue? [y/n]: y

Checking for target database upgrade software version...

BOSTON is running Oracle Version 11.2.0.1

with redo compatibility set to 11.0


reserved.

Phase 2: Continued

Starting SQL Apply on BOSTON ... SQL Apply successfully started.


Estimated catch up time is 20 minutes

Checking DBA_LOGSTDBY_EVENTS for unsupported operations ...

No unsupported operations found

Waiting (will check again in 5 minutes)...





Waiting (will check again in 5 minutes) ...


reserved.

Phase 2: Continued

BOSTON is currently 30 seconds behind CHICAGO

Estimated catch up time is 3 seconds





reserved.

Phase 3: Switchover Phase 3: Switchover to upgraded logical standby

Continue? (y/n): y

Starting switchover sequence: BOSTON will become the new primary

Checking viability of switchover ...

Checked V$DATABASE.SWITCHOVER_STATUS on CHICAGO: ok for switchover

Checked V$ARCHIVE_DEST on CHICAGO: ok for switchover

Checked V$DATAGUARD_STATS on BOSTON: ok for switchover

Sentinel DML reflected on BOSTON: ok for switchover

Switchover operation deemed viable

Stopping services on CHICAGO ...

All services stopped

Disconnecting user sessions on CHICAGO ...

All user sessions disconnected

Switching over CHICAGO to logical standby role ...

CHICAGO is now logical standby


reserved.

Phase 3: Switchover - continued

Switching over BOSTON to the primary database ...

BOSTON is now the primary database

OCI-based services using server-side TAF have been restarted on BOSTON

Shutting down CHICAGO for upgrade ...

shutdown completed



reserved.

Phase 4: Upgrade Original Primary Phase 4: Upgrade the original primary

Continue? (y/n): y

Mounting CHICAGO for flashback ...

Database mounted

Flashing back CHICAGO to Guaranteed Restore Point (ru_primary_before_upgrade)...

Flashback completed

Converting CHICAGO back to a physical standby...

CHICAGO is now a physical standby of BOSTON

Starting media recovery on CHICAGO ...

MRP0 successfully started on CHICAGO

CHICAGO is currently 75 minutes behind BOSTON


Waiting...


reserved.

Phase 4: Upgrade Original Primary - continued CHICAGO is currently 35 minutes behind BOSTON

Estimated catch up time 15 minutes

Checking to see if CHICAGO has mined through upgrade redo...

CHICAGO has now been upgraded automatically to the 11.2.0.1 release

Phase 4 of rolling upgrade completed

Continue to drop all Guaranteed Restore Points and other metadata? {y/n] y

Removing Guaranteed Restore Points ...

Restore points removed

Dropping tables and triggers created as part of rolling upgrade process ...

All supporting tables and triggers dropped

Rolling upgrade process completed.

Current configuration: primary database is BOSTON, standby database is CHICAGO

You can do a switchover if you want to get back to your original configuration.

Rolling Upgrade Workflow will now exit.


reserved.

Benefits of using SQL Apply

• Eliminate application downtime for catupgrd.sql

• Eliminate application downtime due to PL/SQL recompilation

• Method based on Redo Apply only executes one upgrade for the complete Data Guard environment

• Validate the upgraded database on the logical standby

• Fallback for each major step

To Upgrade the Oracle Database Software

Complexity Risk

Same Downtime as Standby First

Patching without any SQL to run! Downtime


reserved.

Program Agenda








reserved.

Upgrading or Migrating with GoldenGate

• Sometimes you just cannot use SQL Apply

– Going between versions that Data Guard does not support.

• 9.2.0.8 to 11.2.0.2

– Platforms combinations that Data Guard does not support.

• Little Endian to Big Endian.

– Moving from non-Oracle to Oracle

• SQL Server to Oracle

– SQL Apply does not yet support a particular data or storage type

• Oracle 10g Transparent Data Encryption to 11g (where it is supported

by SQL Apply)

With Full Data Protection


reserved.

Configure GoldenGate

• Create GoldenGate Target

• Load with an initial extract

• Test New database

• Synchronize

Application Tier

Source

Database (Original Platform,

Version, Source) GoldenGate

Target

Database (New Platform,

Version, Target)


reserved.

Source


Version, Source)

Move Users to Target

Application Tier • Shutdown Source Database

Target


Version, Target)


reserved.

Source


Version, Source)

Move Users to Target

Application Tier • Move users to Target Database

Risk

Why is the Risk Higher?

No Standby for the new environment! Complexity Downtime

Target


Version, Target)


reserved.

Configure for Full Data Protection

Source

Database (Original Platform

and Version)

Target

Database (New Platform

and Version)

Standby


and Version)

Data Guard

Standby


and Version)

Data Guard

GoldenGate


reserved.

When Ready – Shutdown and Move Users

Source


and Version)

Standby


and Version)

Data Guard Data Guard

Target


and Version)

Standby


and Version)

GoldenGate


reserved.

When Ready – Shutdown and Move Users

Much Better!

But what about Fallback? Complexity Risk

Source


and Version)

Standby


and Version)

Downtime


Target


and Version)

Standby


and Version)


reserved.

Restart the Original Standby and Source

Source


and Version)

Target


and Version)

Standby


and Version)

Standby


and Version)



reserved.

Configure GoldenGate to send in reverse

Target


and Version)

Source


and Version)

Standby


and Version)

Standby


and Version)

Fallback always lowers the risk!



GoldenGate


reserved.

Program Agenda








reserved.

Zero Downtime Upgrades or Migrations

• Still some database and application downtime

– To move the users.

• Zero Downtime is Nirvana*

– Remember

• One of these is going to go down.

• Two of these are going to go up.

Complexity Risk

Care to take a guess?

Downtime

*Nirvana can also mean Oblivion


reserved.

Same Starting Point

• We‟re all about reducing Risk

and Protecting Data

• You should already have this

in place.

• If you don‟t shame on you!

Source and Standby on old Platform, Version, etc.

Source


and Version)

Standby


and Version)

Data Guard


reserved.

Prepare for Full Data Protection First

Source


and Version)

Target


and Version)

Standby


and Version)

Standby


and Version)

Data Guard

GoldenGate

Data Guard


reserved.

Enable Zero Downtime!

Source


and Version)

Target


and Version)

Standby


and Version)

Standby


and Version)

GoldenGate

• Set up GoldenGate Bi-Directional Replication

Configure conflict

detection and

resolution Data Guard Data Guard


reserved.

Users Run on both databases

Source


and Version)

Target


and Version)

GoldenGate

Standby


and Version)

Data Guard

Standby


and Version)

Data Guard

Application Tier

Configure conflict

detection and

resolution


reserved.

Users Migrate Over Time

Old Source


and Version)

Target


and Version)

GoldenGate

Standby


and Version)

Data Guard

Standby


and Version)

Data Guard

Application Tier

Configure conflict

detection and

resolution


reserved.

Users Migrate Over Time

Old Source


and Version)

Target


and Version)

Standby


and Version)

Data Guard

Standby


and Version)

Data Guard

Application Tier


reserved.

Bi-Directional Replication

• For Downtime we

achieved zero!

How did we do?

• Complexity goes up due to

Conflict Detection and

Resolution

• Risk increases due to

more moving parts

Downtime

Complexity

Risk


reserved.

Recommendations Best combination for Downtime, Complexity and Risk

Perform on

Standby First

Data Guard

Rolling Upgrade GoldenGate

Hardware Maintenance/Migrations *

Operating System Patches/Upgrades

RAC/ASM Patches

RAC/ASM Upgrades

Oracle Database Standby First Patches

Oracle Database Upgrades *

Zero Downtime Maintenance

* Use Oracle GoldenGate when Data Guard is not supported


reserved.

In Conclusion

• These are bad Complexity Downtime Risk

Downtime Complexity Risk

Complexity Risk

• These are good

• But beware these most of all!

For Doom is written on their dials!


reserved.

Oracle OpenWorld Bookstore

• Visit the Oracle OpenWorld

Bookstore for a fabulous

selection of books on many of the

conference topics and more!

• Bookstore located at Moscone

West, Level 2

• All Books at 20% Discount


reserved.

Resources

• OTN HA Portal:

http://www.oracle.com/goto/availability

• Maximum Availability Architecture (MAA):

http://www.oracle.com/goto/maa

• MAA Blogs:

http://blogs.oracle.com/maa

• Exadata on OTN:

http://www.oracle.com/technetwork/database/exadata/index.html

• Oracle HA Customer Success Stories on OTN:

http://www.oracle.com/technetwork/database/features/ha-casestudies-

098033.html


reserved.

Key HA Sessions, Demos, Labs by Oracle Development

Monday, 3 Oct – Moscone South *

11:00a Auto Detect, Prevent and Repair Data Corruptions, Rm 102

12:30p Future of Oracle Exadata, Rm 104

12:30p RMAN: Not Just for Backups Anymore, Rm 304

2:00p Extreme Data Management, Moscone North Hall D

5:00p Oracle High-Availability System Overview, Rm 104

5:00p GoldenGate Product Update and Strategy, Intercontinental-Sutter

Tuesday, 4 Oct – Moscone South *

10:15a Oracle Secure Backup - Best practices, Rm 304

11:45a Oracle Exadata Technical Deep Dive, Rm 104

3:30p RMAN & Data Guard: Seven Cool Tips from Oracle, Rm 304

3:30p Consolidation on Oracle Exadata, Rm 103

Wednesday, 5 Oct – Moscone South *

10:15a Oracle Active Data Guard - Lessons Learned, Rm 102

1:15p Data Guard for Planned Maintenance, Rm 102

1:15p Understanding Oracle RAC Internals, Rm 103

1:15p Clone Oracle with CloneDB and Direct NFS, Rm 270

Thursday, 6 Oct – Moscone South *

9:00a Exadata Backup and Recovery, Rm 304

10:30a Deduplication and Compression for Backups, Rm 304

12:00p Data Guard Switchover / Failover, Rm 103

3:00p Configure, Size, Monitor Fast Recovery Area, Rm 304

3:00p PeopleSoft with Active Data Guard, Moscone West 2022

Hands-on Labs Marriott Marquis, Salon 14 / 15 Monday, Oct 3, 5:00 pm - 6:00 pm Oracle Active Data Guard

Tuesday, Oct 4, 10:15 am - 11:15 am Oracle Active Data Guard

Demos Moscone South DEMOGrounds

Mon 10:00a - 5:30p Tue 9:45a - 6:00p Wed 9:45a - 4:00p

Maximum Availability Architecture (MAA) Exadata

Active Data Guard Oracle Secure Backup

Recovery Manager & Flashback GoldenGate

Real Application Clusters ASM

*All session rooms at Moscone South unless otherwise noted

*After Oracle OpenWorld, ref. http://www.oracle.com/goto/availability


reserved.

Q&A


reserved.

Latin America 2011 December 6–8, 2011

Tokyo 2012 April 4–6, 2012


reserved.

Powerful New Ways to Use Oracle Data Guard for Planned … · Author: Larry Carpenter Created Date:...

Documents

Transcript of Powerful New Ways to Use Oracle Data Guard for Planned … · Author: Larry Carpenter Created Date:...