Lifecycle Management for Oracle RAC 12c with EMC RecoverPoint

WHITE PAPER

EMC Solutions

Abstract

This white paper examines the advantages of EMC® RecoverPoint® for Oracle RAC life cycle management. It explains RecoverPoint data protection for Oracle RAC and the benefits of using RecoverPoint for applying patches, performing a full Oracle version upgrade, and disaster recovery using remote replication.

July 2014

LIFECYCLE MANAGEMENT FOR ORACLE RAC 12c WITH EMC RECOVERPOINT

Continuous protection for Oracle environments Simple, efficient patch management and failure recovery Minimal downtime for Oracle Database upgrades

2 Lifecycle Management for Oracle RAC 12c with EMC RecoverPoint White Paper

Copyright © 2014 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H13126


Table of contents

Executive summary ............................................................................................................................... 5

Business case .................................................................................................................................. 5

Solution overview ............................................................................................................................ 5

Key benefits ..................................................................................................................................... 5

Introduction.......................................................................................................................................... 7

Purpose ........................................................................................................................................... 7

Scope .............................................................................................................................................. 7

Audience ......................................................................................................................................... 7

Technology overview ............................................................................................................................ 8

Overview .......................................................................................................................................... 8

EMC RecoverPoint ............................................................................................................................ 8

Overview ..................................................................................................................................... 8

RecoverPoint topologies .............................................................................................................. 8

RecoverPoint appliance ............................................................................................................... 9

RecoverPoint splitter ................................................................................................................... 9

Replication sets and consistency groups ..................................................................................... 9

Copy journals, snapshots, and bookmarks ................................................................................ 10

Image access modes ................................................................................................................. 10

Recovery time objective and recovery point objective ................................................................ 11

Oracle Database ............................................................................................................................ 11

Oracle Database 11g Release 2 ................................................................................................. 11

Oracle Database 12c ................................................................................................................. 11

Oracle Real Application Clusters ................................................................................................ 11

VMware vSphere ............................................................................................................................ 12

Solution architecture .......................................................................................................................... 13

Introduction ................................................................................................................................... 13

Architecture overview ..................................................................................................................... 13

Hardware resources ....................................................................................................................... 14

Software resources ........................................................................................................................ 14

Scenario 1: RecoverPoint protection for Oracle environments ............................................................ 15

Introduction ................................................................................................................................... 15

RecoverPoint continuous data protection ....................................................................................... 15

RecoverPoint continuous remote replication .................................................................................. 16

RecoverPoint concurrent local and remote replication .................................................................... 18

Scenario 1: Benefits of RecoverPoint .............................................................................................. 18


Scenario 2: RecoverPoint nondisruptive patch management for Oracle Database .............................. 19

Introduction ................................................................................................................................... 19

Using RecoverPoint for Oracle RAC patch management .................................................................. 19

Overview ................................................................................................................................... 19

Step 1: Create the consistency group ........................................................................................ 20

Step 2: Create a pre-patch, application-consistent bookmark on the production server ............. 21

Step 3: Enable the bookmark on the remote server.................................................................... 21

Step 4: Apply and test the patch on all the RAC nodes on the remote server .............................. 22

Step 5: Apply the patch on the production server ...................................................................... 23


Scenario 3: RecoverPoint process for upgrading Oracle Database 11g R2 to Oracle Database 12c .... 25

Introduction ................................................................................................................................... 25

Using RecoverPoint to upgrade Oracle Database ............................................................................ 25

Overview ................................................................................................................................... 25

Step 1: Set up RecoverPoint replication (CRR) from the source to the target ............................... 26

Step 2: Create a database instance on the target ....................................................................... 26

Step 3: Upgrade the target instance .......................................................................................... 26

Step 4: Restore a RecoverPoint snapshot from the source to the target ..................................... 28

Step 5: Upgrade the Data Dictionary to Oracle Database 12c ..................................................... 28

Step 6: Test the target instance ................................................................................................. 28

Step 7: Switch users to the target .............................................................................................. 28

Step 8: Set up RecoverPoint replication from target to source .................................................... 29

Step 9: Upgrade the original source instance ............................................................................ 29


Conclusion ......................................................................................................................................... 30

Summary ....................................................................................................................................... 30

Findings ......................................................................................................................................... 30

References.......................................................................................................................................... 31

EMC documentation ....................................................................................................................... 31

Oracle documentation.................................................................................................................... 31


Executive summary

Oracle Real Application Clusters (RAC) environments are among the most visible and mission-critical in daily use by EMC customers. For this reason, Oracle RAC environments have rigorous recovery point objective (RPO) and recovery time objective (RTO) requirements. Typically, Oracle RAC 11g and Oracle RAC 12c environments are replicated to a remote disaster recovery site to prevent data loss in the event of a site-wide disaster. This is because Oracle databases frequently contain the source business data for the enterprise, and this data must be protected at all costs. EMC® RecoverPoint® provides a simple, reliable, efficient, and high-performance remote replication solution for Oracle RAC.

Oracle frequently releases patches for Oracle RAC. Applying these patches can be disruptive to Oracle RAC applications. RecoverPoint bookmarks provide a simple, efficient, and reliable mechanism for patching Oracle RAC on a remote copy of the environment, testing the patches, and then propagating the patches to the production site.

With the transition from Oracle RAC 11g to Oracle RAC 12c now underway, customers need a solution that enables a major Oracle version upgrade with minimum downtime. RecoverPoint can significantly reduce the production downtime needed for upgrading Oracle RAC, compared with other replication tools.

This solution provides use case scenarios that an Oracle database administrator (DBA) can relate to a typical Oracle RAC production environment setup. The use cases demonstrate the advantages of using EMC RecoverPoint.

For the solution, an Oracle database is deployed as a virtualized RAC cluster on EMC VNX® storage and is replicated to VNX storage at a remote site. RecoverPoint appliances (RPAs) manage all aspects of data replication, and array-based splitters in the VNX ensure that the RPAs receive every write for replication.

This white paper demonstrates the benefits of RecoverPoint for various purposes, including:

Continuous data protection:

RecoverPoint provides continuous data protection both locally and remotely for applications running in heterogeneous storage and infrastructure environments. A dedicated, out-of-band RecoverPoint appliance (RPA) manages all aspects of data replication with minimal impact to an application’s I/O operations.

RecoverPoint provides data protection against data corruption, human errors, infrastructure failure, and local or regional disasters.

RecoverPoint provides minimal application downtime during planned server or storage upgrades.

RecoverPoint protects all business application data continuously in addition to the database.

Business case

Solution overview

Key benefits


Ease of applying patches and recovering from failures:

With RecoverPoint, you can easily roll back a local or remote copy of your data to any point-in-time and recover an application-consistent image of your Oracle RAC database in the event of a failure.

RecoverPoint’s ability to roll back to any point-in-time also facilitates easy and efficient deployment of Oracle RAC and application patches.

Minimal downtime for upgrading Oracle RAC software:

The solution demonstrates how to use RecoverPoint to upgrade Oracle RAC 11g to Oracle RAC 12c with minimal production downtime.


Introduction

This white paper documents the benefits of EMC RecoverPoint for remote replication of Oracle RAC environments, and for patching and upgrading production Oracle RAC databases. The paper describes three customer use cases and the advantages of using RecoverPoint in each case.

This white paper demonstrates the following customer use cases:

Scenario 1: Business continuity and disaster recovery

Scenario 2: Oracle Database patch management

Scenario 3: Oracle Database version upgrades

This white paper is intended for Oracle DBAs, storage administrators, virtualization administrators, system administrators, and managers of these professionals, as well as any others involved in evaluating, managing, maintaining, acquiring, or operating Oracle RAC database environments.

Purpose

Scope

Audience


Technology overview

The solution uses these technologies:

EMC RecoverPoint

Oracle Database

Oracle RAC

VMware vSphere

Overview

EMC RecoverPoint is an advanced data protection, replication, and disaster recovery solution that is designed with the performance, reliability, and flexibility required for enterprise applications in heterogeneous storage and server environments. It provides local replication and bidirectional remote replication for physical, virtual, and cloud environments.

RecoverPoint topologies

As shown in Figure 1, RecoverPoint supports local replication, remote replication, and concurrent local and remote replication.

Figure 1. RecoverPoint replication options

RecoverPoint continuous data protection (CDP)—CDP intercepts every write to the production volumes and replicates them to a local copy. This enables both any point-in-time recovery and a complete restoration of the production volumes with no data loss. CDP supports both synchronous and asynchronous replication.

Overview

EMC RecoverPoint


RecoverPoint continuous remote replication (CRR)—CRR intercepts every write to the production volumes and replicates them to a remote copy over any distance. CRR supports synchronous and asynchronous replication over a Fibre Channel or IP connection. CRR also supports bi-directional replication.

Synchronous replication provides any point-in-time recovery and a zero RPO. Asynchronous replication provides recovery to specific points in time, with a minimal RPO.

RecoverPoint concurrent local and remote replication (CLR)—CLR is a combination of CRR and CDP and provides concurrent local and remote data replication.

CDP is normally used for operational recovery, and CRR is used for disaster recovery. CLR is used to provide both local and remote protection for the same production volumes.

RecoverPoint appliance

RecoverPoint is appliance-based, which enables it to better support large amounts of information stored across heterogeneous environments. The out-of-band approach enables RecoverPoint to deliver continuous replication without affecting an application’s I/O operations.

RecoverPoint appliances (RPAs) run the RecoverPoint software and manage all aspects of data replication. For local replication, a cluster of two or more active RPAs is deployed—this supports immediate switchover to another appliance if one of the RPAs in a cluster goes down. For remote replication, a RecoverPoint cluster is deployed at both sites.

For remote replication, the RPAs use powerful deduplication, compression, and bandwidth reduction technologies to minimize the use of bandwidth and dramatically reduce the time lag between writing data to storage at the source and target sites. For local replication and synchronous remote replication, the time lag is zero.

RecoverPoint splitter

RecoverPoint uses lightweight, write-splitting technology, on the application server, in the fabric, or in the array, to mirror application writes to the RPA. This solution uses the array-based RecoverPoint splitter that is built into the EMC VNX storage system. The RecoverPoint splitter intercepts and splits each application write, sending a copy first to the RPA and then to the designated storage volume.

Replication sets and consistency groups

RecoverPoint uses replication sets and consistency groups to ensure that updates to the production volumes1 are written to the copy volumes in consistent write order. A replication set defines an association between a production volume and any local or remote copy volumes to which it is replicating. A consistency group logically groups replication sets that must be consistent with one another. The consistency group

1 Production volumes are the volumes to which the host applications write data and that RecoverPoint is protecting. Copy volumes are the target volumes to which RecoverPoint replicates the production volumes.


ensures the write-order consistency of all copies so that you can always use them to continue working or to restore the production source.

RecoverPoint protection is policy-driven. A replication policy, based on a particular business need, can be uniquely specified for each consistency group. This policy governs the replication parameters for the consistency group—for example, the RPO and RTO for the consistency group, its bandwidth reduction settings, and its replication mode (synchronous or asynchronous).

Copy journals, snapshots, and bookmarks

RecoverPoint maintains a copy journal for each local and remote copy to which it is replicating a consistency group. These journals are dedicated to holding point-in-time snapshots, which are collections of one or more writes to production storage. These snapshots enable the copy storage to be rolled back to a previous point in time (PIT).

When replicating synchronously, each write is a snapshot, enabling recovery to any point in time. When replicating asynchronously, the RPA groups writes several before distributing them to the copy journals—this supports recovery to significant points in time.

Each snapshot in a journal includes only data that has changed from the previous snapshot. That is, a snapshot is the difference between one consistent image of the stored data and the next. All snapshots are crash consistent by default.

RecoverPoint uses bookmarks to label particular snapshots for easy identification. Bookmarks are useful for marking specific points in time for later recovery or failover.

Image access modes

To test a copy to verify that it is a reliable and consistent copy of the production storage image, it is necessary to access the image of the copy. Image access is also required to restore the production storage from the copy, to roll back to a previous state of the data, to recover files, and to fail over to the copy.

When replicating normally, the state of the storage at the copy site is No access; that is, a host cannot access the storage. When image access is enabled for a particular snapshot, host applications at the copy site can access and work with the copy.

RecoverPoint provides the following image access modes:

Logged Access—Recover Point rolls snapshots back or forward, applying each in turn to the stored image until the selected point in time is reached.

Virtual access—RecoverPoint creates the image on a virtual disk by pulling the needed data either from snapshots or from storage. The virtual disk is deleted when image access is disabled.

Direct access—Host applications write directly to the copy storage.

Disabling image access restores the storage state to No access and restores the copy to the state it was in before it was accessed.


Recovery time objective and recovery point objective

The recovery time objective (RTO) is the duration of time and a service level within which a business process must be restored after a disaster or disruption. The RTO defines the time required to enable physical access to the latest application-consistent image at a copy site.

The recovery point objective (RPO) is the point in time to which a business process must be restored after a disaster or disruption. The RPO defines how much data it is acceptable to lose in a disaster situation.

For example, if a company’s data must be restored to within 30 minutes of a disaster event and the time it takes to get the recovered data back into production is 6 hours:

The RPO is 30 minutes

The RTO is 6 hours

If you require an RPO of zero, you must configure RecoverPoint to replicate in synchronous mode—that is, to replicate each write to the copy site before the host can initiate another write.

Oracle Database 11g Release 2

Oracle Database 11g R2 is available in a variety of editions tailored to meet the business and IT needs of an organization. This solution uses Oracle Database 11g R2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. Oracle Database 11g R2 EE supports advanced features, such as Virtual Private Database, and data warehousing options such as partitioning and advanced analytics.

Oracle Database 12c

Oracle Database 12c is the latest generation Oracle database. It is designed for the cloud, and provides a new multitenant architecture that simplifies the process of consolidating databases onto the cloud. Oracle Database 12c includes many enhancements and new features, including Oracle Multitenant, Automatic Data Optimization, security innovations, and new and enhanced high availability features. Oracle Enterprise Manager 12c Cloud Control enables administrators to easily implement and manage new Oracle Database 12c functionality and the new multitenant architecture.

Oracle Real Application Clusters

Oracle Real Application Clusters (RAC) is an option to the Oracle Database EE. Oracle RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide highly scalable and available database solutions. Oracle RAC is a key component of Oracle’s private cloud architecture.

Oracle Database


VMware vSphere is the virtualization platform used by thousands of IT environments around the world. vSphere abstracts applications and information from the complexity of the underlying infrastructure through comprehensive virtualization of server, storage, and networking hardware. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications just like physical computers. The virtualization of hardware resources enables efficiencies through consolidation of multiple applications on fewer physical servers.

VMware vCenter is a scalable and extensible platform for centralized management of vSphere environments, providing control and visibility at every level of the virtual infrastructure.

VMware vSphere


Solution architecture

This solution demonstrates the use of EMC RecoverPoint for protecting Oracle databases, and for Oracle Database patch management and full version upgrades. A VMware virtualized Oracle Real Applications Clusters (RAC) database is deployed on VNX5500™ storage. The RecoverPoint system consists of RecoverPoint splitters built into the VNX5500 arrays and RecoverPoint appliance clusters at the production and recovery sites.

Note: While the solution uses VNX5500 storage arrays, you can use any VNX or VMAX array that is configured appropriately to provide the storage.

Figure 2 shows the physical architecture of the solution.

Figure 2. Solution architecture

Introduction

Architecture overview


The architecture consists of the following layers:

Compute—At each site, the compute layer comprises two servers and an RPA cluster. The servers at each site are configured as a VMware ESXi cluster. Two virtual machines created on the ESXi cluster form a two-node Oracle RAC database. A third virtual machine runs the management console.

Network—The network layer comprises two IP switches and dual-speed, director-class SAN switches, which we2 configured to produce 10 and 8 Gb/s active bandwidth. The SAN switches are designed for deployment in storage networks that support virtualized data centers and enterprise clouds.

Storage—A VNX5500 storage array, with integrated array-based RecoverPoint splitter, provides the storage at each site.

Table 1 details the hardware environment for the solution.

Table 1. Solution hardware resources

Compute hardware Quantity Configuration

EMC RecoverPoint appliances 4 Gen 4: 2 RPAs per site

EMC VNX5500 storage systems 2 1 VNX5000 array per site

VMware ESXi servers 4 Eight cores, 3.3 GHZ processors

500 GB RAM

Fibre Channel (FC) switches 4 8 Gb/s FC

Ethernet switches 4 1Gb/s and 10 Gb/s

Table 2 details the software components of the solution’s compute layer.

Table 2. Solution software resources

Compute software Version Description

EMC RecoverPoint 4.0.SP2.P1 EMC RecoverPoint software installed on RPAs

VNX Block Operating Environment

05.32.000.5.201 Includes EMC Unisphere Block, EMC RecoverPoint splitter

EMC Unisphere 1.2.25.1.0156 VNX management

Oracle ASMLib 2.0 Support Library for ASM

Oracle RAC 11g R2 (Grid infrastructure)

Enterprise Edition 11.2.0.2

Oracle Clusterware and ASM

Oracle RAC 12c (Grid infrastructure)

Enterprise Edition 12.1.0.1.0

Oracle Clusterware and ASM

Oracle Enterprise Linux 6.4 Operating system for database servers

VMware vSphere 5.5 Hypervisor hosting all virtual machines

2 In this paper, "we" refers to the EMC Solutions engineering team that validated the solution.

Hardware resources

Software resources


Scenario 1: RecoverPoint protection for Oracle environments

RecoverPoint provides a single solution for business continuity and disaster recovery in heterogeneous infrastructure and application environments. The out-of-band RecoverPoint appliance (RPA) can replicate data both locally and remotely to enable recovery of Oracle database environments from data corruption, human errors, infrastructure failure, and local or regional disasters.

RecoverPoint tracks every write sent to the production database storage volumes, as follows:

With CDP, RecoverPoint continuously replicates the writes to a local copy as well as distributing them to the production volumes. RecoverPoint also maintains a local journal of all changes, which enables roll back to, and recovery from, any point in time.

With CRR, RecoverPoint continuously replicates database writes to a remote copy at the recovery site and maintains a remote journal of all changes. This enables complete recovery of the Oracle database and point-in-time rollback from the remote copy.

CRR enables data replication across completely separate fabrics over distances limited by IP technology or FC distance extension technology.

The combination CDP and CRR enables concurrent local and remote replication (CLR).

Critically, RecoverPoint consistency groups ensure the consistency and write-order fidelity of the local and remote copies and the point-in-time history, so you can always use the copies to restore the production database or to continue working. In addition, recovery of the local or remote copy does not affect ongoing replication or data protection.

Note: See also the EMC white paper Simplified Oracle RAC 12c Backup, Cloning, and Protection for Oracle DBAs - Using VMware vSphere 5.5, EMC RecoverPoint, EMC Data Domain, EMC VNX, EMC VNX Snapshots, and EMC VNX SnapView Clones.

With CDP, the RecoverPoint splitter sends a copy of each write to the production LUNs and the RecoverPoint Appliance (RPA). The RPA writes the data to the local journal with a time stamp and bookmark metadata. The RPA then distributes the data to the local copy in a write-order-consistent manner.

Figure 3 illustrates the data flow for asynchronous CDP with the VNX write splitter. All production volumes are VNX volumes.

Introduction

RecoverPoint continuous data protection


Figure 3. Continuous data protection (CDP)

The data flow for asynchronous CDP is as follows:

1. The host issues a write to a LUN that is protected by RecoverPoint. The VNX splitter intercepts the write.

2. The splitter “splits” the write and sends it to the production volume and the local RPA.

3. When the RPA receives the write, it immediately returns an ACK to the splitter. (With synchronous replication, the RPA delays the ACK until the data has been written to the copy journal.) The LUN also acknowledges the write.

4. The RPA writes the data to the copy journal at the production site.

5. The RPA distributes the write to the local copy; write order is preserved during distribution. If a consistency group contains many LUNs, all the data written to the local copy is write-order consistent.

With CRR, the RecoverPoint splitter sends a copy of each write to the production LUN and the local RecoverPoint Appliance (RPA). The RPA processes the data, including applying bandwidth reduction techniques according to pre-defined policies. The RPA then transmits the data to the remote RPA, which uncompresses the data and distributes it to the remote journal volume and the remote copy with write-order consistency.

RecoverPoint continuous remote replication


Figure 4 illustrates the data flow for asynchronous CRR with the VNX write splitter. All production volumes are VNX volumes.

Figure 4. Continuous remote replication (CRR)

The data flow for asynchronous CRR is as follows:

1. The host issues a write to a LUN that is protected by Recover Point. The VNX splitter intercepts the write.

2. The splitter “splits” the write and sends it to the production volume and the local RPA.

3. When the RPA receives the write, it immediately returns an ACK to the splitter. (With synchronous replication, the RPA delays the ACK until the data has been written to the remote journal). The LUN also acknowledges the write.

4. When the write is received by the local RPA, it is bundled with other writes, deduplicated to remove redundant blocks, sequenced, and time stamped. The local RPA then compresses the package and transmits it with a checksum for delivery to the remote RPA cluster.

5. When the remote RPA receives the package, it verifies the checksum to ensure that the package was not corrupted in transmission, and uncompresses the data.

6. The RPA then writes the data to the journal at the remote site.

7. After the data has been written to the journal volume, it is distributed to the remote copy volumes. Write order is preserved during this distribution.


Figure 5 Illustrates the data flow for asynchronous CLR with the VNX write splitter. All production volumes are VNX volumes. The data flow is a combination of the CDP and CRR data flows, with RecoverPoint performing local and remote replication simultaneously for the same production volumes.

Figure 5. Concurrent local and remote replication (CLR)

RecoverPoint is a compelling solution for customers with Oracle environments when compared to similar replication tools. The benefits of RecoverPoint include the following:

A significant amount of a customer’s business-critical data resides outside the Oracle RAC database environment: in operating system files, non-database files, non-Oracle DBMSs, and file systems. RecoverPoint can replicate your entire Oracle application infrastructure, including across federated environments.

You can use RecoverPoint in heterogeneous application and infrastructure environments where a single replication solution is desired to reduce costs and simplify management.

Where network bandwidth cost is a primary concern and compression of all data transmitted over the network is a requirement, RecoverPoint can be an effective solution. The bandwidth reduction features of RPAs can reduce bandwidth requirements by up to 90%.

With RecoverPoint, the bandwidth used for remote replication can be throttled.

RecoverPoint CRR optimizes CPU use on the production server by offloading data transfer to the remote site to the RPAs.

RecoverPoint synchronous replication provides any point-in-time recovery and a zero RPO. RecoverPoint asynchronous replication provides recovery to specific points in time, with a minimal RPO.

RecoverPoint concurrent local and remote replication

Scenario 1: Benefits of RecoverPoint


Scenario 2: RecoverPoint nondisruptive patch management for Oracle Database

EMC RecoverPoint enables you to apply Oracle RAC patches and minor version upgrades to production databases with minimal downtime or disruption to user applications. If required, you can also roll back patches by restoring a pre-patch copy of the database on the production server.

With RecoverPoint, you can clone a production copy of the database on a remote server without affecting production applications. You can then use standard Oracle utilities to test the patch on the remote copy before applying the patch to the production environment. Replication of the production database continues uninterrupted during testing and there is no data loss. With testing completed, you can apply the patch to the production environment during a scheduled maintenance window without affecting the production database.

RecoverPoint tracks all database writes in history journals at the production site, remote site, or both. The journals hold time-stamped snapshots of the production database that make it possible to recover the database to any point in time. You can bookmark (or label) snapshots to provide easily identifiable recovery points, such as a pre-patch recovery point. If there is a problem with a patch or minor version upgrade, you can quickly and easily restore the database to a pre-patch state from a bookmark or snapshot.

RecoverPoint application-consistent, point-in-time recovery capabilities optimize the RPO and RTO for Oracle data.

For this use case scenario, RecoverPoint is replicating data from the production site to a remote site using CRR in asynchronous mode. The Oracle database files and redo logs are part of the same RecoverPoint consistency group to ensure that updates to the production database are always written to the remote copy in consistent and correct write order.

Overview

To apply a patch using RecoverPoint, follow these high-level steps:

1. Create a RecoverPoint consistency group for the database files and redo logs and start replicating.

2. Create a pre-patch bookmark on the production server.

3. Enable the bookmark on the remote server.

4. Apply and test the patch on all the RAC nodes on the remote server.

5. Apply the patch on the production server.

Introduction

Using RecoverPoint for Oracle RAC patch management


Step 1: Create the consistency group

In Unisphere for RecoverPoint:

1. Use the Protect Volumes wizard to create a consistency group for the database files and log files.

2. Specify CG_ORA_RP_PROD1 as the name of the consistency group and define the replication sets shown in Figure 6.

Note: Figure 6 shows the Manage Protection screen, which enables you to view and modify the attributes of a selected consistency group. In this case, CG_ORA_RP_PROD1 is selected and the right pane shows the replication sets for that consistency group.

Figure 6. Viewing and modifying consistency groups

3. The default replication mode for a consistency group is Asynchronous replication mode. Do not change this.

4. When you finish the Protect Volumes wizard, RecoverPoint starts replicating the consistency group to the remote site.

For details of all consistency group settings, refer to the EMC RecoverPoint 4.0 Administrator’s Guide.


Step 2: Create a pre-patch, application-consistent bookmark on the production server


1. Select Protection > Manage Protection and select the Production copy in the CG_ORA_RP_PROD1 consistency group.

2. Select the Apply Bookmark option, as shown in Figure 7, and specify a name such as oracle_upgrade_testing as the bookmark name.

Figure 7. Bookmark and test a copy

Step 3: Enable the bookmark on the remote server

1. Test the bookmarked image at the remote site to verify that it is a reliable and consistent copy of the production database.


a. Select Recovery > Test a Copy to start the Test a Copy wizard; select the oracle_upgrade_testing bookmark on the ORA_RP_DR server for testing.

b. Click Next Test the Image. RecoverPoint starts enabling image access to the copy volumes.

When replicating normally, the state of the storage at the remote site is No Access, as shown in Figure 7. When RecoverPoint has enabled image access, the status changes to Logged Access.

2. Mount the database and enable ASM on the remote server:

a. Use ESXi shell to list the LUNs on the ORA_RP_DR server:

#esxcli storage vmfs snapshot list


b. Resignature and add the LUNs to the VMFS datastore:

#esxcli storage vmfs snapshot resignature –l $volumes

The ESXi servers on the remote site can now see the datastore.

c. Add disks on the ORA_RP_DR server. You can do this manually by using the vSphere Client or automatically by using a script.

d. Scan and mount the Oracle ASM disk groups DATA, FRA, REDO, and CRS.

3. Test the database applications on the ORA_RP_DR server:

a. Start the Oracle database on ORA_RP_DR with the startup command. Use the sysdba account.

b. Use SQL *Plus or a user application to test the database user accounts, and create a report for comparison with the results of the post-patch test.

Step 4: Apply and test the patch on all the RAC nodes on the remote server

1. Apply the Oracle patch or minor version upgrade on the ORA_RP_DR server using Oracle Database Upgrade Assistant (DBUA) or the Oracle OPatch utility.

Note: Refer to the Critical Patch Updates, Security Alerts and Third Party Bulletin on the Oracle website for the latest Oracle Database patches.

a. Select a different ORACLE_HOME directory and create a link to the default Oracle home.

b. Use the standard Oracle Deployment Procedure or the relevant Readme to patch the database.

c. Use the All Nodes method to patch all Oracle RAC nodes at the same time, as shown in Figure 8.

Figure 8. Oracle RAC All Nodes patching

http://www.oracle.com/technetwork/topics/security/alerts-086861.html


In the example in Figure 8, systems A, B, C, and D, are nodes in a 4-node RAC cluster. When you perform All Nodes patching on the cluster, nodes A, B, C, and D are brought down, the patch is applied to each node in turn, and all the nodes are then brought back up again.

2. Test the database applications after applying the patch:

a. Use SQL *Plus or a user application to test the database applications and create a report.

b. Verify the report against the report generated before you applied the patch.

3. Disable image access on the ORA_RP_DR server:

a. Unmount the volumes and close access to the snapshot on the ORA_RP_DR server.

This operation discards all write operations on the server.

b. Delete the disks on the ORA_RP_DR server. You can do this manually by using the vSphere Client or automatically by using a script.

Step 5: Apply the patch on the production server

1. Schedule down time and apply the patch on the production server (ORA_RP_PROD). Use DBUA to apply the patch:

a. Before you bring down the production database, flush the source transactions from the RPAs at the production site to the remote site.

b. Gracefully shut down the production database for host activities. This ensures an up-to-date image of the database file system. Flush all file systems that reside on the production volumes.

c. Select a different ORACLE_HOME directory and create a link to the default Oracle home.

d. Use the standard Oracle Deployment Procedure to apply the patch.

2. Test the database applications after patching the production server:

a. Use SQL *Plus or a user application to test the database applications and create a report.

b. If there is a problem with the patch or upgrade, restore the database from a pre-patch snapshot.

3. Bring up all the production applications.


The advantages of RecoverPoint for patch management include the following:

RecoverPoint offers continuous remote replication with any point-in-time recovery. You can use snapshots or bookmarks to easily restore production on any remote servers by mounting the snapshot volumes on the servers. There is no impact to the production server while testing Oracle patches on the remote server.

RecoverPoint is designed to protect critical business processes and improve operational recovery with minimal impact to production environments. You can restore snapshots and bookmarks on the remote site for test purposes while replication is running.

RecoverPoint preserves a record of all write transactions and provides crash- or application-consistent recovery points to any point in time. This means that, if data is lost or corrupted because of server failure, software errors, or user errors, it is always possible to recover a clean copy of the affected data.

With RecoverPoint, you can recover a database at either the local or the remote sites. The recovery points can be immediately accessed and mounted back to the production environment to perform recovery without the need to restore from a backup.

RecoverPoint can protect all applications and data on the storage array, not just Oracle databases.

RecoverPoint provides a single solution for testing application patches; the patch testing process is not limited to Oracle software.



Scenario 3: RecoverPoint process for upgrading Oracle Database 11g R2 to Oracle Database 12c

To upgrade your Oracle RAC production environment with a major new release, you can take advantage of RecoverPoint continuous replication to upgrade the production database with minimal interruption to the business. In this use case scenario, we upgrade from Oracle Database 11g to Oracle Database 12c, but you can use the same procedure to upgrade from an earlier version.

For this use case scenario, RecoverPoint is replicating data from the production site (source) to the remote site (target) using CRR in asynchronous mode. The source site is running Oracle RAC 11g.

To test the Oracle RAC 12c upgrade, you create an instance of the production database on the target site and run all the necessary application tests on the remote servers.

To upgrade the production site, reverse the source and target roles, and migrate all production activities to the target site. RecoverPoint switches to replicating data from the target Oracle RAC 12c database to the source Oracle RAC 11g database. The source instance remains available for failback if necessary. When the production database is successfully running on the target, you can upgrade the source without affecting users, who will be working on the target site while the upgrade is in progress. When the source has been successfully upgraded and tested, you migrate all production activities back to the source site and switch back to replicating from the source to the target.

Overview

For migrating to Oracle Database 12c from a previous version, we recommend the following procedure:

1. Set up RecoverPoint replication (CRR) from the source to the target.

2. Create a database instance on the target.

3. Upgrade the target instance.

4. Restore a RecoverPoint snapshot from the source to the target.

5. Upgrade the Data Dictionary to Oracle Database 12c.

6. Test the target instance thoroughly from platform validation through to application validation.

7. Switch users to the target.

8. Set up RecoverPoint replication from the target to the source.

9. Upgrade the original source instance.

Introduction

Using RecoverPoint to upgrade Oracle Database


Step 1: Set up RecoverPoint replication (CRR) from the source to the target

Before you create a database instance at the target site to test the upgrade, you need to set up RecoverPoint CRR replication between the source and the target to ensure that there is an adequate buildup of snapshots in the copy journal for successful recovery and failover. This requires you to configure a consistency group that contains the source and target copies of the database files and log files. Replication also needs to be in progress for performance testing on the upgraded database to determine if infrastructure upgrades or changes to RecoverPoint parameters are necessary to achieve the same throughput as with Oracle RAC 11g.

Step 2: Create a database instance on the target

For testing the upgrade, you need to create an instance of the production database at the target site. Many methods are available for doing this, both physical and logical. RecoverPoint bookmarks and disk mirroring (split off a mirrored copy) provide the easiest methods. Mount the copy on the same mount point on the target system and open the instance. If you need to create a logical database, use Oracle Data Pump to export and import the data.

Step 3: Upgrade the target instance

To upgrade the target instance to Oracle Database 12c, use Oracle Installer and Oracle Upgrade Assistant Utility (DBUA). Perform the upgrade according to Oracle instructions. If logical instantiation is required because of platform differences or restructuring of the new database, create an empty Oracle Database 12c database and use Data Pump Export and Import to load the data to the target database.

Figure 9 through Figure 11 show a selection of Oracle Installer and DBUA pages from the process for upgrading Oracle Database 11g R2 to Oracle Database 12c.

Figure 9. Oracle Installer – Select Installation Option


Figure 10. Database Upgrade Assistant – Network Configuration

Figure 11. Database Upgrade Assistant – Database Upgrade Summary and Upgrade Results

Summary of upgrade options before the upgrade starts

Summary of upgrade results


Step 4: Restore a RecoverPoint snapshot from the source to the target

With RecoverPoint, you can restore a database on the target site without creating an additional copy of the database and without interfering with the replication process.

Use RecoverPoint image access to restore a bookmark taken while the Oracle database was running. Restore the snapshot by using the RecoverPoint CLI or the Unisphere UI to enable image access.

Step 5: Upgrade the Data Dictionary to Oracle Database 12c

With the database binaries upgraded to Oracle Database 12c, and the refreshed data catalog at Oracle Database 11.2, you need to upgrade the catalog to 12c.

Start the database in upgrade mode and run the following catalog upgrade scripts:

Cd $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/perl/bin/perl catctl.pl catupgrd.sql

Step 6: Test the target instance

Thorough testing of the upgrade involves many types of tests, including:

Performing preliminary checks on the target server

Running read-only tests and checking reports and queries for Oracle Database 12c compatibility

Implementing the desired Oracle Database 12c features

Verifying that applications, both packaged and custom, are compatible with the new database

Follow these steps:

1. Plan the tests: Make a list of the applications that require testing. Organize application testing based on the importance or complexity of the applications. Make sure to include special periodic applications, such as month-end and quarter-end processes.

2. Run a test: To test applications that update the data in the target database instance, first stop the replicating process.

3. Reset the test environment after each test: Restore the database from the RecoverPoint snapshot whenever it is necessary to bring the database back to the testing baseline.

4. When you have completed application testing, restore the latest production snapshot of the database using the information in following steps of the upgrade process: Step 4: Restore a RecoverPoint snapshot from the source to the target and Step 5: Upgrade the Data Dictionary to Oracle Database 12c.

Step 7: Switch users to the target

To upgrade the source site without affecting users, transfer all user activity to the new database at the target site—that is, the production Oracle RAC 11g database that has been restored at the target site. With RecoverPoint replicating data from the Oracle Database 11g system on the source site, you can fail back to that site if there are problems with the upgraded environment.


Step 8: Set up RecoverPoint replication from target to source

After switching users on the target database, use Unisphere to set up RecoverPoint CRR in the opposite direction—that is, from Oracle Database on the target to Oracle Database on the source.

Step 9: Upgrade the original source instance

When the production environment is running successfully on Oracle Database 12c at the target site, upgrade the source system from Oracle Database 11g to Oracle Database 12c. This process is much faster and easier than on the target site, due to the lessons learned while upgrading that site. Repeat the process used to perform the upgrade on the target site. At this point, RecoverPoint is replicating data from the original target to the original source.

The advantages of using RecoverPoint for upgrading to Oracle Database 12c from a previous version include the following:

For a hardware, operating system, or Oracle Database upgrades, using RecoverPoint to build a replica server for testing purposes can deliver significant time and cost savings when compared to using Oracle RMAN backup.

RecoverPoint enables you to test an upgrade on a database copy without affecting production. RecoverPoint replication with best practices ensures the migration of data to the new environment without data loss and without disrupting users. Replication from production to copy databases is not affected during the upgrade process.

With RecoverPoint, an upgrade is not performed directly on production and users do not switch to the test environment until the upgrade there has been fully tested. This approach minimizes risk and disruption to production users.



Conclusion

This solution demonstrates the advantages and benefits of using EMC RecoverPoint for remote replication of Oracle RAC databases, and for patching and upgrading Oracle RAC software.

The benefits of EMC RecoverPoint include:

RecoverPoint offloads the remote replication workload from the production RAC servers onto the RPAs, thereby eliminating the performance impact of remote replication on the production RAC cluster.

RecoverPoint includes advanced WAN bandwidth reduction capabilities. In environments where WAN bandwidth is expensive, the savings on WAN costs alone can justify implementation of a RecoverPoint solution.

RecoverPoint provides minimal production RAC node downtime when applying Oracle patches, which improves the stability and performance of the Oracle RAC cluster.

When migrating to a new major version of Oracle Database, the DBA needs to clone a copy of the production database, and upgrade and test that copy. RecoverPoint bookmarks provide an ideal solution for this process.

Summary

Findings


References

The following documents, available on the EMC Online Support or EMC.com websites provide additional and relevant information. If you do not have access to a document, contact your EMC representative.

EMC Recover Point: Replicating VMware Technical Notes

EMC RecoverPoint: Replicating Oracle with RecoverPoint Technical Notes

EMC RecoverPoint Release 4.0 Administrator’s Guide

EMC VNX Family: VNX5500 Hardware Information Guide

Simplified Oracle RAC 12c Backup, Cloning, and Protection for Oracle DBAs - Using VMware vSphere 5.5, EMC RecoverPoint, EMC Data Domain, EMC VNX, EMC VNX Snapshots, and EMC VNX SnapView Clones

Refer to My Oracle Support for the following Oracle documents:

Oracle Real Application Clusters Installation Guide 11g Release 2 (11.2) for Linux and Unix

How to migrate an existing pre 12c database (non-CDB) to 12c CDB database (Doc ID 1564657.1)

Backup and Recovery Scenarios (Doc ID 94114.1)

Oracle Database Administrator’s Guide: 12c Release 1 (12.2)

Oracle Enterprise Manager Lifecycle Management Administrator's Guide 12c Release 4 (12.1.0.4)

EMC documentation

Oracle documentation

https://support.emc.com/

http://www.emc.com/resource-library/resource-library.htm

https://support.oracle.com/

Lifecycle Management for Oracle RAC 12c with EMC RecoverPoint

Documents

Transcript of Lifecycle Management for Oracle RAC 12c with EMC RecoverPoint