Oracle database with RAC 12c HA BP TCOUG

1/16/14

1

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1

Oracle Database with Real Application Clusters (RAC) 12c High Availability Best Practices Markus Michalewicz Director of Product Management Oracle Real Application Clusters (RAC)

@OracleRACpm http://www.linkedin.com/in/markusmichalewicz

http://www.slideshare.net/MarkusMichalewicz

1/16/14

2


Agenda

§  High Availability and Oracle’s MAA

§  Oracle RAC 12c and Standardization

§  Bronze Level High Availability

§  Silver Level High Availability

§  Gold Level High Availability

§  Summary and Q&A

§  Appendices A-B


Reduce cost and increase return on investment

High Availability (HA) Business Challenges

Eliminate risk of downtime and data loss

1/16/14

3


Generic volume manager & file system…

Generic backup software

Generic cold failover cluster

Failover Server Production Server

Identical storage

Idle DR

Storage mirroring

Traditional Availability Cold Start = High Risk Idle Assets = High Cost


Edition-based Redefinition, Online Redefinition, Data Guard, GoldenGate –  Minimal downtime maintenance, upgrades, migrations

Active Data Guard – Data Protection, DR – Query Offload

GoldenGate – Active-active replication – Heterogeneous

Active Replica

Oracle Maximum Availability Architecture (MAA)

RMAN, Oracle Secure Backup – Backup to disk, tape or cloud

Enterprise Manager Cloud Control – Coordinated Site Failover Application Continuity – Application HA Global Data Services – Service Failover / Load Balancing

RAC – Scalability – Server HA

Flashback – Human error

correction

Production Site

ASM – ASM mirroring

1/16/14

4


Applying MAA Principles

1.  Assess impact of downtime & data loss

2.  Define service level objectives –  Recovery Time Objective (RTO):

§  How long can you afford to be down?

–  Recovery Point Objective (RPO): §  How much data can you afford to lose?

–  Performance: §  pre and post failure

Begin with a business impact analysis

RAC

GoldenGate

RMAN Data Guard


Applying MAA Principles A simplified two-step implementation approach

Standardize

Set of reference HA architectures

Reduce costs

Consolidate

1/16/14

5


Oracle Database 12c MAA

Three Standard Reference Architectures

BRONZE Minutes to days of downtime Data protected as of last backup

SILVER Seconds to minutes of downtime Near-zero data loss

GOLD Zero application outage Zero data loss


BRONZE

1/16/14

6


SILVER


GOLD

1/16/14

7


Agenda









Standardize on Clusters Clusters increase scalability and HA regardless of DB types used later

Oracle GI

Oracle GI

Oracle GI for a Cluster

Oracle GI

for a

Cluster

High Availability

Scal

abili

ty

License Costs

Oracle GI

Standalone

Oracle GI Re-configuration

to enable cluster

1/16/14

8


Standardize on Oracle RAC For all the flexibility you need

Oracle GI DB A DB A

Oracle GI DB A DB A

DB A

Oracle RAC One Node Oracle RAC

Flexibility

Scal

abili

ty

Online

upgrade

DB A

Oracle Restart

Oracle GI Online (rolling) Patch and PSU Application


Standardize on Oracle RAC and Oracle Multitenant The new standard for Oracle Database Consolidation

Oracle GI

Oracle RAC One Node

Consolidation

Agi

lity

Oracle GI

Oracle RAC

1/16/14

9

17

Commonwealth Bank n  The Commonwealth Bank is one of Australia’s leading providers of integrated financial

services including retail, business and institutional banking, funds management, superannuation, insurance, investment and broking services. The Bank is one of the largest listed companies on the Australian Stock Exchange.

18

Introduction

n  In 2007 CBA set out to create an Oracle database shared service offering for the bank •  The offering has been highly successful by several measures •  Oracle-as-a-Service has continued to be developed through several iterations

n  Oracle as a Service (OaaS) v1 – went live May 2008

1/16/14

10

19

n  Host many Oracle database applications on a cluster of hardware

n  Processor consolidation

•  Run each server hotter •  Take advantage of complimentary workload peaks.

n  Higher Availability

•  Load balancing •  HA failover for component failure •  Standby DR •  Most apps do not implement these features – too expensive

n  Cost Reduction

n  Better Service

•  Full time experts •  Always on-call

n  Reduced Risk

•  Whole environment is managed •  Operated as a “business”

Oracle as a Service In A Nutshell

20

Reduce Risk, Improve Time to Market n  For new Projects:

•  Remove a phase from the project – infrastructure already in place •  Remove reliance on expensive/scarce SME resources for design and build •  No longer need to manage risk associated with procurement and build •  Time to instantiate a new Production quality environment: 3 months -> 2 minutes.

n  Example: New ISV Application introduced into our Online Share Trading platform •  Required to test performance under the workload

& data volume conditions projected in 2 years time.

Dedicated Infrastructure OaaS

Implementation Time 3-4 months few hours

$ Cost to Project Several hundred thousand < $10K

On Project Completion Under-utilized asset remains Environment turned-off

1/16/14

11


Agenda









BRONZE

•  Minimize the cost of HA –  Use HA features included with Oracle Database –  Utilize backups to protect against media and site failures –  Secure offsite tape storage (in the cloud) for DR

•  Optionally –  Consolidate with Oracle Multitenant –  Improve HA with RAC One Node –  Self-Service provisioning with

Enterprise Manager Database as a Service

Bronze Level High Availability

1/16/14

12


BRONZE

Physical Data Protection Data Corruption Protection & Oracle ASM (Mirroring)

Human Error Correction Flashback Technologies: Drop, Query, Transaction, Table, and Database

Planned Maintenance Online Redefinition Online Reorganization Edition Based Redefinition Online File Move Online Patching

Fast Backup & Recovery Recovery Manager (RMAN) Fast Recovery Area Oracle Secure Backup

High Availability at the Bronze Tier Features Integrated with the Oracle Database


Oracle Database Inherent Protection

§  Oracle Data blocks have a well-defined structure –  Block header is kept

consistent with payload –  Enables validation of

both physical and logical intra-block consistency

§  Oracle ensures block validity is maintained as it traverses I/O path

–  Extensive corruption checks: §  DB_BLOCK_CHECKSUM: http://docs.oracle.com/cd/E11882_01/server.112/e40402/initparams049.htm

§  DB_BLOCK_CHECKING: http://docs.oracle.com/cd/E18283_01/server.112/e17110/initparams046.htm

§  For more information see My Oracle Support (MOS) note 1302539.1

Data Corruption Protection

1/16/14

13


§  ASM supports ALL data – database files, file systems, Clusterware files (OCR, Voting Disk) §  Built-in mirroring protects from disk failures §  Auto-repair of corrupt blocks using a valid mirror copy

3rd Party FS Application

Automatic Storage Management

ASM Cluster & Single Node File System (ACFS)

Database

ACFS Snapshot

ASM Disk Group

DB Datafiles, OCR and Voting Files Oracle Binaries 3rd Party File Systems

Dynamic Volume Manager

ASM Instance Managing Oracle DB Files

Oracle Automatic Storage Management (ASM)


Small ASM Features – Big Difference Disk and disk group operations

Failure Group repair time

§  Similar to existing disk repair time

§  Default is 24 hours

§  Disk group attribute: failgroup_repair_time

Disk Resync §  Accepts power limit

§  Range 1 (least resources) to 1024 (most resources)

§  Checkpoints: failed operations restart at the last checkpoint

Fast Disk Replacement §  Efficient, atomic operation

§  No disk group reorganization required

Data Scrubbing §  Check for logical consistency

(and repair from mirror)

§  during rebalance.

§  areas on demand

§  Protects seldom-accessed data

1/16/14

14


§  Fast point-in-time recovery (PITR) without expensive restore operation

§  Error investigation –  View data as of previous point in time

§  Error correction –  Back-out a transaction –  Incorrect table updates –  Rewind the entire database

§  Rolling upgrades, Snapshot Standby

@T2 Col-1 Col-.. Col-n

Row-1 tom 1234 vp

Row-2 ben 8834 vp

Row-3 charlie 9837 vp

Row-n tom 8793 vp

@T1 Col-1 Col-.. Col-n

Row-1 abby 1234 officer

Row-2 ben 8834 mgr

Row-3 Charlie 9837 officer

Row-n tom 8793 vp Wrong Update

Flashback Table

DB @ T1 DB @ T2

Batch Update

Flashback Database

Wrong Update

Flashback Technologies Rewind Button for Oracle Databases


§  Unique knowledge of database file formats and recovery procedures

–  Oracle block validation –  Online block-level recovery –  Native encryption, compression –  Table/partition-level recovery –  Oracle Multitenant support

§  Tape and cloud backups §  Unified Management

Oracle Enterprise Manager

Tape Drive

Oracle Public Cloud Amazon S3

RMAN

Data Files Fast Recovery

Area (FRA)

Ora

cle

Secu

re

Bac

kup

Oracle Recovery Manager (RMAN) Database Integrated Backup and Recovery

1/16/14

15


1 0 1 0 1 0 1 0 1 1 1 0 1 0 0 1 1 0 1 1 0 1 1 1 1 0 0 0 1 0 1 0 0 0 0 1 0 0 1 1 0 1 1 0 1 0 0 1

§  Simple RECOVER TABLE command to recover one or more tables (most recent or older version) from an RMAN backup

§  Eliminates time and complexity associated with manual restore, recover & export

–  Enables fine-grained point-in-time recovery of individual tables instead of the contents of the entire tablespace

RMAN Backups

Fine-grained Table Recovery From Backup using a simple RMAN command


1 1 1 0 1

1 0 1 1 0 1 1 1 1 0 1

0 0 1 0 1 0 0 0 0 1

1 0 0 1

1 1 1 0 1 1 0 1 1 0 1 1 1 1 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 1

Cross-Platform Backup & Restore

§ Simplifies procedure for platform migration § Minimize read-only impact with multiple incremental backups

Simplified Platform Migration

Source Database (AIX)

Backup to Disk/Tape

(data files, optional endian conversion, metadata export)

Restore Backup (optional endian

conversion, metadata import)

Destination Database (Solaris)

1/16/14

16


BRONZE

•  Optionally –  Consolidate with Oracle Multitenant –  Improve HA with RAC One Node –  Self-Service provisioning with

Enterprise Manager Database as a Service

•  Benefit: –  Simplified consolidation è Lower risk

–  “Cloud Consolidation with Oracle RAC – How much is too much?” can be simpler

–  “Manage many as one”

Bronze Level High Availability


Oracle Multitenant Architecture Components of a Multitenant Container Database (CDB)

Pluggable Databases (PDBs)

PDBs

Root

CDB

1/16/14

17


Oracle Multitenant Architecture – Dynamics

§  PDBs share common SGA and background processes

§  Foreground sessions see only the PDB they connect to


Advantages of Oracle Multitenant Architecture Reduced CapEx & OpEx, Increased Agility, Easy Adoption

Self-contained PDB for each application §  Applications run unchanged §  Rapid provisioning (via clones) §  Portability (via pluggability)

Shared memory and background processes §  More applications per server

Common operations performed at CDB level §  Manage many as one (upgrade, HA, backup) §  Granular control when appropriate

1/16/14

18


Consolidate and Protect Data RMAN and Flashback with Oracle Multitenant

•  RMAN backup/restore –  One backup protects a CDB and all of its PDBs –  PDBs can also be backed up in isolation

•  RMAN point in time recovery –  CDB: all PDBs recovered to same point in time –  PDBs: individual PDB can be recovered

with no impact on other PDBs

•  Clone CDB and PDB level •  Flashback at CDB level


Agenda








1/16/14

19


SILVER

Silver Level High Availability

•  Bronze, plus: •  Server and instance HA •  RAC One Node •  Oracle RAC

•  Database and site HA/DR •  Active Data Guard •  GoldenGate •  Site Guard


Standardize on Clusters Clusters increase scalability and HA regardless of DB types used later

Oracle GI

Oracle GI

Oracle GI for a Cluster

Oracle GI

for a

Cluster

High Availability

Scal

abili

ty

License Costs

1/16/14

20


Oracle Automatic Storage Management (ASM) 12c


Oracle Automatic Storage Management (ASM) Oracle Database 11.2 or earlier configuration

ASM Cluster Pool of Storage

Node2

Disk Group B Disk Group A

Node3 Node5 Node4

Shared Disk Groups Wide File Striping

One to One Mapping of ASM Instances to Servers

Node1

ASM Instance

Database Instance

ASM Disk

RAC Cluster

DBA DBA DBB DBB DBC DBB

ASM ASM ASM ASM ASM

1/16/14

21


Oracle ASM 12c – Overview Oracle ASM 12c Standard Deployment


Disk Group B Disk Group A Shared Disk Groups Wide File Striping

One to One Mapping of ASM Instances to Servers

ASM Instance

Database Instance

ASM Disk

RAC Cluster

Node4 Node3 Node2 Node1 Node5 ASM ASM ASM ASM ASM

ASM Instance

Database Instance

DBA DBA DBC DBB DBB DBB


Introducing Oracle Flex ASM Removal of One to One Mapping and HA



Databases share ASM instances

ASM Instance

Database Instance

ASM Disk

RAC Cluster

Node5 Node4 Node3 Node2 Node1

Node5 runs as ASM Client to Node4




ASM ASM ASM

ASM Instance

DBA DBA DBC DBB DBB DBB

More Information in Appendix A

1/16/14

22


Supporting Pre-Oracle 12c Databases Pre-Oracle 12c Databases require a local ASM instance



Databases share ASM instances

ASM Instance

Database Instance

ASM Disk

RAC Cluster

Node5 Node4 Node3 Node2 Node1 ASM ASM ASM

DBA DBA DBC

ASM ASM

11gDB

11gDB DBB DBB DBB


Oracle RAC One Node 12c Oracle RAC One Node 12c provides:

1.  Best in-class single instance Oracle Database availability

2.  Better database consolidation

3.  Better database virtualization

Using

§  A standardized and improved deployment and management

§  A familiar and matured HA stack –  Oracle Grid Infrastructure

Oracle Grid Infrastructure (GI):

Automatic Storage Management (ASM)

Oracle Clusterware (OCW)

Oracle Flex ASM

CRM

Oracle RAC

One Node A

Oracle RAC

One Node B

1/16/14

23


Oracle RAC One Node Architecture §  Oracle RAC One Node is

–  an Oracle RAC enabled database –  with only one instance running

under normal operation

§  In case of a server or instance failure, the one running instance is started on another server in the cluster (failover)

§  Online Database Relocation (Oracle RAC One Node specific feature) minimizes downtime during scheduled maintenance operations.

Basic features

Oracle GI DB A

DB A

DB B

DB B


Oracle RAC One Node vs. Oracle RAC “Always On” vs. “Always Running”

Oracle GI DB A

DB A

DB B

DB B

Oracle GI

DBA

DBA

DBB

DBB

§  Oracle RAC: multiple instances per database running concurrently

§  Oracle RAC One Node: one instance per database running at a time

1/16/14

24


Database Failover

§  On failure of –  a database (DB) instance –  Or the server hosting the DB –  Or other failures impacting operation

§  Oracle RAC One Node will fail over the database instance to another server in the cluster

§  Servers can be chosen using either administrator-managed or policy-managed resource allocation rules

Oracle RAC One Node provides standard DB cold-failover functionality

Node3 Node2

Oracle (Flex) ASM based pool of shared storage

Node1

Public Network

DBB DBC DBA

Oracle Grid Infrastructure

DBE DBD DBA


Online Database Relocation for better HA

§  Online Database Relocation minimizes downtime during (scheduled) maintenance operations such as:

–  Patching

–  Resource re-allocation due to demand

Oracle RAC One Node-specific functionality that makes the difference

Node3 Node2

Oracle (Flex) ASM based pool of shared storage

Node1

Public Network

DBB DBC


DBE DBD DBA

DBA Online DB Relocation

1/16/14

25


Online Database Relocation For no downtime patching and other maintenance operations

Node3 Node2 Node1

Public Network


DBE DBD DBA DBC

DB connections

DB Service

DB Home

DBB


Online Database Relocation Step 1: Start target DB instance and relocate DB Service

Node3 Node2 Node1

Public Network


DBE DBD

DB connections

DB Service

DBA DBA Online DB Relocation

DBC

1/16/14

26


Online Database Relocation Step 2: Stop source DB instance transactional

Node3 Node2 Node1

Public Network


DBE DBD DBA Online DB Relocation DBA

DB connections

DB Service

New Connection

DBC


Online Database Relocation Step 3: Patch Oracle Database home on the source server

Node3 Node2 Node1

Public Network


DBE DBD DBA

+1

DB connections

DB Service

Patch +1

DBC

1/16/14

27


Online Database Relocation Step 4: Rewind (relocate back) to activate patch usage

Node3 Node2 Node1

Public Network

DBC


DBE DBD DBA

DB connections

DB Service

Patched DB Home


Online Database Relocation Step 5: Patch remaining home(s)

Node3 Node2 Node1

Public Network


DBE DBD DBA DBC DBB

DB connections

DB Service

Patched DB Home

1/16/14

28


Online Database Relocation Optimization: 3-step patching for consolidated environments

Node3 Node2 Node1

Public Network


DBE DBD DBA DBC

DB connections

DB Service

DB Home

DBB

DB connections

DB Service

Patched DB Home

1 Patch Target

2 Relocate

3 Patch remaining

DBA DBB


The New Oracle RAC 12c Oracle RAC 12c provides:

1.  Better Business Continuity and High Availability (HA)

2.  Agility and Scalability

3.  Cost-effective Workload Management

Using

§  A standardized and improved deployment and management

§  A familiar and matured HA stack

Oracle Real Application Clusters (RAC)

Oracle Grid Infrastructure (GI):

Automatic Storage Management (ASM)

Oracle Clusterware (OCW)

Oracle Flex ASM

CRM

1/16/14

29


Oracle RAC Support for Oracle Multitenant


Oracle Multitenant on Oracle RAC Consider a Single Instance (SI), non-CDB

Server

Database Instance

Services

Server

Database Instance

CRM North

CRM South Services CRM

Reporting

1/16/14

30


Oracle Multitenant on Oracle RAC Then consider a RAC Database, non-CDB

Node 1

RAC Instance 1

Services

Node 1

RAC Instance 1

CRM North

CRM South Services

Node 2

RAC Instance 2

CRM Reporting


Oracle Multitenant on Oracle RAC Finally, picture a CDB RAC Database

Node2

CDB Instance 2

Node1

CDB Instance 1

CDB

Services

1/16/14

31


Oracle Multitenant on Oracle RAC The simplest way of converting a SI PDB to RAC: unplug/plug

Node2

CDB Instance 2

Node1

CDB Instance 1

CDB


Use Services with Policy Managed Databases For better HA and workload management

§  Oracle RAC-based systems allow for management of services based on policies.

§  Server Pools are used to group workload to run on either one or a group of servers in the pool.

§  Using policies allows for adapting to changes in demand (Dynamic Database Services)

HRg1wc HR PDB

CRM PDB CRMSVC

vixen srvpool: backoffice comet srvpool: frontoffice

dancer srvpool: frontoffice dasher srvpool: frontoffice Oracle GI for a cluster Oracle GI for a cluster

Oracle GI for a cluster Oracle GI for a cluster

Oracle RAC Oracle RAC


raccdb1

raccdb1_1

raccdb1_4 Raccdb1_3

Raccdb1_2

1/16/14

32


Policy-Managed Databases with Oracle RAC 12c Improved HA Management - Startup

raccdb1

§  Defining a Service Startup Order –  Ensure services are started in

specified groups and specified order. –  Leverages Server Pool

Minimum and Importance properties

dasher srvpool: frontoffice Oracle GI for a cluster

Oracle RAC

raccdb1_4

1st

vixen srvpool: frontoffice Oracle GI for a cluster

Oracle RAC

raccdb1_3

2nd

comet srvpool: backoffice Oracle GI for a cluster

Oracle RAC

raccdb1_1

4th

dancer srvpool: backoffice Oracle GI for a cluster

Oracle RAC

raccdb1_2

3rd



Oracle RAC

raccdb1_2

dancer srvpool: frontoffice Oracle GI for a cluster

Oracle RAC

raccdb1_2

Policy-Managed Databases with Oracle RAC 12c Improved HA Management – New Failover Strategy

raccdb1


Oracle RAC

raccdb1_4


Oracle RAC

raccdb1_3


Oracle RAC

raccdb1_1

§  Servers “Move” to Replace Failed node –  Protects against cascade failures due to load –  Ensures workload isolation between pools –  Less important workloads

shutdown transactional

1/16/14

33



Oracle RAC

raccdb1_2

dancer srvpool: frontoffice Oracle GI for a cluster

Oracle RAC

raccdb1_2


Oracle RAC

raccdb1_1

comet srvpool: frontoffice Oracle GI for a cluster

Oracle RAC

raccdb1_1

Policy-Managed Databases with Oracle RAC 12c Improved HA Management – Manage Last Service Standing

raccdb1


Oracle RAC

raccdb1_4


Oracle RAC

raccdb1_3

§  Business Critical Services survive multiple failures –  Most important pool always gets the servers

–  Services preserved across multiple failures

–  Less important workloads shutdown transactional

More Information in Appendix B


Oracle RAC and Virtualization

1/16/14

34


Private Database Cloud Architectures Oracle Database 12c

Dedicated Databases

share servers and OS

Virtual Machines

share servers

Pluggable Databases

share servers, OS and database

Increasing Consolidation


Virtualization – a Black Box Approach A simple solution without insight

Bare-Metal Server

DOM-0 Hypervisor Guest

Dom-0 Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0 Guest Guest

1/16/14

35


Live Migration = Moving a Black Box Live Migration moves a black box from one server to another

Bare-Metal Server


Dom-0 Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0 Guest Guest

Live Migration


Virtual Machine HA = Restarting a Black Box Virtual Machine HA restarts / fails over a virtual machine on failure

Bare-Metal Server


Dom-0 Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0 Guest Guest

1/16/14

36


Virtual Machine HA = Restarting a Black Box But what if the application needs attention?

Bare-Metal Server


Dom-0 Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0 Guest

DBA ?


High Availability Requires Insight The Oracle RAC Stack provides the insight

Bare-Metal Server


Dom-0 Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0 Guest Guest

DBC1 DBA1

+1

DBC2 DBA2

+1

1/16/14

37


Agenda









GOLD

•  Silver, plus: •  Zero application outage

•  Application Continuity

•  Zero data loss over WAN •  Active Data Guard Far Sync

•  Zero downtime maintenance •  Oracle GoldenGate

•  Automated workload management for replicas •  Global Data Services

Gold Level High Availability

1/16/14

38


Dealing with Unknown Transaction State

CRM

DB A DB A

DB A DB A

DB A

Pre-Oracle Database 12c § Database outages can cause

in-flight work to be lost, leaving users and applications in doubt

§ Often leads to –  User pains –  Duplicate submissions –  Rebooting mid-tiers –  Developer pains


Tackling In-Flight Work for Unplanned Outages

§ Transaction Guard –  A reliable protocol and API that returns the outcome of the last transaction

§ Application Continuity –  Safely attempts to replay in-flight work following outages and planned

operations.

Two New Features

1/16/14

39


Application Continuity

CRM

Masks Unplanned & Planned Outages

DB A DB A

DB A DB A

DB A

§  Replays in-flight (DML) work on recoverable errors

§  Masks many hardware, software, network, storage errors and outages when successful

§  Improves end-user experience and productivity without requiring custom application development ✔


Application Continuity – Example

Application Servers

Database Servers

End User

A reliable replay of in flight work

Network Switches

User selects product from application and purchases it from the web checkout

User transaction arrives at application infrastructure. It makes it’s way through the application tiers and results in a database transaction being created

1/16/14

40



Application Servers

Database Servers

End User


Network Switches The infrastructure hosting the database fails just before the transaction is committed to the database.



Application Servers

Database Servers

End User


Network Switches

The jdbc driver detects the failure and checks with an available node in the cluster, using “Transaction Guard”, whether the transaction committed or needs to be replayed

If the transaction needs to be replayed, “Application Continuity” will submit all of the inflight work to a surviving node in the cluster and perform a commit. This all happens transparently to the application

1/16/14

41



Application Servers

Database Servers

End User


Network Switches

The user receives confirmation that his order has been successfully completed.


Agenda








1/16/14

42


BRONZE SILVER GOLD

RTO = Minutes to days RTO = Seconds to minutes Zero application outage

RPO = Since last backup RPO = Near zero Zero data loss

Minimize CapEx Minimize OpEx

•  Scalable performance •  No single point of failure •  Real-time data protection

•  Fast failover •  Minimal planned downtime

•  High ROI

•  Zero application outage •  Zero data loss at any distance

•  Zero downtime for maintenance •  Automated workload

management for replicated databases

MAA Deployment Architectures – Local HA Address the Complete Range of Enterprise HA Requirements


Questions and

Answers

1/16/14

43


Agenda









Appendix A: Oracle Flex ASM – Details and Test Scenarios

1/16/14

44


Oracle Flex ASM Configuration details

§  Flex ASM introduces new local resources: –  At least one ASM listener

–  One “proxy_advm” (per node)

§  Used for ACFS access to Flex ASM instances

§  Connections from a database instance to an ASM instance are based on SQLnet using listeners.

–  The listener directs the connection to the least loaded ASM instance based on the load metric it maintains.

–  The connection details are fetched from CSS global data

–  The ASM instance to which the database instance connects to is listed in the database alert log:

§  NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10004)

–  The userid and password supplied are also managed automatically. They are supplied while establishing the session, not while connecting.

[GRID]> crsctl stat res -t ------------------------------------------------------------------------------- Name Target State Server State details ------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------

ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE ONLINE ONLINE vixen STABLE ... ora.DATA.dg ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE OFFLINE OFFLINE vixen STABLE ... ora.INFRA.dg ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE OFFLINE OFFLINE vixen STABLE ...

ora.proxy_advm ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE ONLINE ONLINE vixen STABLE


Oracle Flex ASM Administering Flex ASM

§  Check ASM mode using asmcmd

§  Check where ASM instances are running using srvctl

§  Check the configuration of ASM using srvctl

§  Modify the ASM configuration using srvctl

§  Cluster ASM listener: –  Flex ASM can have its own network or overlap with the

private interconnect network used by RAC DB instances

–  It is recommended to configure a full overlap of the private interconnect and the ASM network in current use cases:

[GRID]> asmcmd ASMCMD> showclustermode ASM cluster : Flex mode enabled

[GRID]> srvctl status asm ASM is running on comet,dasher,dancer

[GRID]> srvctl config asm -detail ASM home: /u01/app/12.1.0/grid Password file: +INFRA/orapwASM ASM listener: LISTENER ASM is enabled. ASM instance count: 3

Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM

[GRID]> srvctl modify asm -count all

[GRID]> srvctl status asm ASM is running on comet,dasher,dancer

[GRID]> srvctl start asm

[GRID]> srvctl status asm ASM is running on comet,vixen,dasher,dancer

1/16/14

45


Oracle Flex ASM Some test scenarios…

§  A failure of an ASM instance is without effect for the databases using ASM disk groups (here: -MGMTDB)

–  If an ASM instance is running on all nodes (count all) the ASM instance is restarted upon failure

–  If an ASM instance is running on a subset of nodes the ASM instance might be restarted on another node

–  The client using the ASM instance may change

§  A failure of an ASM instance is without effect for the APX (ADVM proxy) running on the same node

§  A failure of the APX will lead to a restart of the database instance (on the node) that uses an ACFS file system.

§  More information: –  http://docs.oracle.com/cd/E16655_01/server.121/e17612/

asminst.htm#OSTMG95329

[GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4093 1 0 11:35 ? 00:00:00 asm_pmon_+ASM1 grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4 [GRID]> kill -9 4093 [GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4

[GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4 grid 5748 1 0 11:39 ? 00:00:00 asm_pmon_+ASM1 grid 5793 19013 0 11:39 pts/1 00:00:00 grep pmon

[GRID]> asmcmd lsct data DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX1 DATA

[GRID]> ssh vixen /u01/app/12.1.0/grid/bin/asmcmd lsct data DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX2 DATA +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX3 DATA


Appendix B: Policy-Managed Databases with Oracle RAC 12c

1/16/14

46


Policy-Managed Databases with Oracle RAC 12c Database Services

vixen srvpool: frontoffice

dancer srvpool: backoffice dasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB

Oracle GI | HUB


Oracle RAC

raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

[GRID]> srvctl status serverpool Server pool name: frontoffice Active Servers count: 2 Server pool name: backoffice Active Servers count: 2 [RAC]> srvctl status service -d raccdb1 Service crmsvc is running on nodes: dasher,vixen Service hrsvc is running on nodes: comet,dancer

§  Database Services –  Ensure that workload is hosted in the respective

server pool with the cardinality defined as part of the policy definition.

comet srvpool: backoffice Oracle GI | HUB

Oracle RAC

raccdb1_1


Policy-Managed Databases with Oracle RAC 12c Improved HA Management - Startup

raccdb1

[GRID]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 Server pool name: Free Importance: 0, Min: 0, Max -1

§  Defining a Service Startup Order –  Ensure services are started in specified groups

and specified order.

–  Leverages Serverpool Min and Importance properties

dasher srvpool: frontoffice Oracle GI | HUB

Oracle RAC

raccdb1_4

1st

vixen srvpool: frontoffice Oracle GI | HUB

Oracle RAC

raccdb1_3

2nd


Oracle RAC

raccdb1_1

4th

dancer srvpool: backoffice Oracle GI | HUB

Oracle RAC

raccdb1_2

3rd

1/16/14

47



Oracle RAC

raccdb1_2

dancer srvpool: frontoffice Oracle GI | HUB

Oracle RAC

raccdb1_2

Policy-Managed Databases with Oracle RAC 12c Improved HA Management – New Failover Strategy

raccdb1

[GRID]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [GRID]> srvctl status serverpool Server pool name: frontoffice Active Servers count: 2 Server pool name: backoffice Active Servers count: 1


Oracle RAC

raccdb1_4


Oracle RAC

raccdb1_3


Oracle RAC

raccdb1_1

§  Servers “Move” to Replace Failed node –  Protects against cascade failures due to load

–  Ensures workload isolation between pools

–  Less important workloads shutdown transactionally



Oracle RAC

raccdb1_2


Oracle RAC

raccdb1_2

Policy-Managed Databases with Oracle RAC 12c Improved HA Management – Manage Last Service Standing

raccdb1

[RAC]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [RAC]> srvctl status service –db Service crmsrv is running on nodes comet,vixen Service hrsvc is not running


Oracle RAC

raccdb1_4


Oracle RAC

raccdb1_3


Oracle RAC

raccdb1_1

§  Business Critical Services survive multiple failures –  Most important pool always gets the servers

–  Services preserved across multiple failures

–  Less important workloads shutdown transactionally

comet srvpool: frontoffice Oracle GI | HUB

Oracle RAC

raccdb1_1

1/16/14

48


Policy-Managed Databases with Oracle RAC 12c Improved HA Management – Dynamic Provisioning

vixen srvpool: frontoffice


Oracle GI | HUB


Oracle RAC

raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

[GRID]> srvctl modify serverpool –serverpool backoffice –max 1 [RAC]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 4 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [RAC]> srvctl status service –db Service crmsrv is running on nodes dasher,dancer,vixen Service hrsvc is not running


Oracle RAC

raccdb1_1

§  Add Servers Just-In Time to meet demand –  Server Pools sized via Min and Max properties

–  Dynamically controlled by QoS Management

–  Planned control via Clusterware or QoSM Policies


Oracle RAC

raccdb1_2


Policy-Management with Oracle RAC 12c Support for Multiple Policies tracking business objectives

§  More Information: –  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/pbmgmt.htm

§  New in Oracle Grid Infrastructure 12c –  Server Categories

§  Server Categories use server attributes to allow for an active use of differently sized servers

–  Policy Sets §  Policy Sets allow for dynamic adjustment

to demand changes in an atomic transaction

Server Categories •  NAME •  ACL •  EXPRESSION •  …

Server Attributes •  NAME •  MEMORY_SIZE •  CPU_COUNT •  CPU_CLOCK_RATE •  CPU_HYPERTHREADING •  CPU_EQUIVALENCY • …

[GRID]> crsctl modify policyset –attr "LAST_ACTIVATED_POLICY=NightTime“ CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded CRS-2673: Attempting to start 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2672: Attempting to start 'ora.raccdb1.hrsvc.svc' on 'dancer‘

…

QoS Management

OR Oracle Clusterware - CRSCTL

1/16/14

49


Policy-Management with Oracle RAC 12c Setting up policy sets – Provision Server Pools and creating a PolicySet

§  Add another server pool “backup”

§  Set up policy set with 3 server pools & 3 policies as follows: –  DayTime:

§  frontoffice uses three servers (MIN_SIZE=3)

§  backoffice uses one server (MIN_SIZE=1)

§  backup does not run during daytime (MIN_SIZE=0)

–  NightTime:

§  frontoffice uses one server (MIN_SIZE=1)

§  backoffice uses two servers (MIN_SIZE=2)

§  backup uses only one server (MIN_SIZE=1)

–  Weekend:

§  frontoffice uses one server (MIN_SIZE=1)

§  backoffice uses one server (MIN_SIZE=1)

§  backup uses two servers (MIN_SIZE=2)

[GRID]> srvctl add serverpool –serverpool backup –min 0 –max 2 –importance 20 [GRID] srvctl status serverpool Server pool name: frontoffice Active Servers count: 3 Server pool name: backoffice Active Servers count: 1 Server pool name: backup Active Servers count: 0

[GRID]> crsctl modify policyset -file /u01/app/…/policyset.txt


Policy-Management with Oracle RAC 12c Using Policy Sets – part 1: Activate a policy

§  Activate the policy for the weekend

[GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=Weekend" [GRID]> srvctl config serverpool Server pool name: Free Importance: 0, Min: 0, Max: -1 Category: Candidate server names: Server pool name: Generic Importance: 0, Min: 0, Max: -1 Category: Candidate server names: Server pool name: backoffice Importance: 0, Min: 1, Max: 1 Category: Candidate server names: Server pool name: backup Importance: 1, Min: 2, Max: 2 Category: Candidate server names: Server pool name: frontoffice Importance: 0, Min: 1, Max: 1 Category: Candidate server names:

1/16/14

50


Policy-Management with Oracle RAC 12c Using Policy Sets – part 2: check the result

vixen srvpool: frontoffice comet srvpool: backup

dancer srvpool: backup dasher srvpool: backoffice Oracle GI | HUB Oracle GI | HUB

Oracle GI | HUB Oracle GI | HUB



raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1

[RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer Instance raccdb1_3 is running on node vixen Instance raccdb1_4 is running on node dasher [RAC]> srvctl status service -d raccdb1 Service backup is running on nodes: comet,dancer Service crmsvc is running on nodes: vixen Service hrsvc is running on nodes: dasher

§  With an activated weekend policy


Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – part 1

§  Assume you have 2 servers that have better IO –  Use these servers for backups whenever possible

§  Here comet and dancer have better IO by definition

§  What you need to do: –  Set up a server category that identifies the servers

–  Add the use of the server category to the server pool §  Define the server pools that utilize the category

and during which policy activation it shall be used.

§  You need to restart the cluster stack on the servers that you modify in this fashion

[GRID]> su Password: [GRID]> crsctl set server label IOplus ... #On dancer [GRID]> crsctl set server label Ioplus [GRID]> crsctl get server label CRS-4972: Current SERVER_LABEL parameter value is Ioplus [GRID]> crsctl status server comet dancer –f Comet Dancer

NAME=comet MEMORY_SIZE=3338 CPU_COUNT=1 CPU_CLOCK_RATE=2132 CPU_HYPERTHREADING=0 CPU_EQUIVALENCY=1000 DEPLOYMENT=other CONFIGURED_CSS_ROLE=hub RESOURCE_USE_ENABLED=1 SERVER_LABEL=IOplus ...

NAME=dancer MEMORY_SIZE=3338 CPU_COUNT=1 CPU_CLOCK_RATE=2247 CPU_HYPERTHREADING=0 CPU_EQUIVALENCY=1000 DEPLOYMENT=other CONFIGURED_CSS_ROLE=hub RESOURCE_USE_ENABLED=1 SERVER_LABEL=IOplus ...

1/16/14

51


Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – part 2

§  What you need to do: –  Set up a server category that identifies the servers -

done –  Add the use of the server category to the server pool

§  Define the server pools that utilize the category and during which policy activation it shall be used

§  Note: Add the server category usage to the server pool(s) that require it & specify the policies that should be altered accordingly

–  In this example the backup server pool is the one that will benefit the most and the policies will only start this server pool as required.

[GRID]> crsctl add category moreIO –attr "EXPRESSION='SERVER_LABEL co IOplus'" [GRID]> crsctl status category moreIO NAME=moreIO ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTIVE_CSS_ROLE=hub EXPRESSION=(SERVER_LABEL co IOplus) [GRID]> crsctl modify serverpool ora.backup -attr "SERVER_CATEGORY=moreIO" -all_policies


Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – comparison

Before After

[GRID]> crsctl status policy DayTime POLICY NAME=DayTime DESCRIPTION= SERVERPOOL NAME=Free ... SERVERPOOL NAME=ora.backoffice IMPORTANCE=0 MAX_SIZE=1 MIN_SIZE=1 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.backup IMPORTANCE=0 MAX_SIZE=0 MIN_SIZE=0 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.frontoffice IMPORTANCE=0 MAX_SIZE=3 MIN_SIZE=3 SERVER_CATEGORY= SERVER_NAMES=

[GRID]> crsctl status policy DayTime POLICY NAME=DayTime DESCRIPTION= SERVERPOOL NAME=Free ... SERVERPOOL NAME=ora.backoffice IMPORTANCE=0 MAX_SIZE=1 MIN_SIZE=1 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.backup IMPORTANCE=0 MAX_SIZE=0 MIN_SIZE=0 SERVER_CATEGORY=moreIO SERVER_NAMES= SERVERPOOL NAME=ora.frontoffice IMPORTANCE=0 MAX_SIZE=3 MIN_SIZE=3 SERVER_CATEGORY= SERVER_NAMES=

1/16/14

52


Policy-Management with Oracle RAC 12c Using Policy Sets and the effect of Server Categories






raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1


dancer srvpool: frontoffice dasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB




raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1


[GRID]> crsctl modify policyset –attr "LAST_ACTIVATED_POLICY=DayTime“ [GRID] srvctl status serverpool Server pool name: frontoffice Active Servers count: 3 Server pool name: backoffice Active Servers count: 1 Server pool name: backup Active Servers count: 0

[RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dasher,vixen,comet Service hrsvc is running on nodes: dancer

Policy-Management with Oracle RAC 12c Using Policy Sets means changing policies on a push of a button

raccdb1


Oracle RAC

raccdb1_3


Oracle RAC

raccdb1_4

comet srvpool: backup Oracle GI | HUB

Oracle RAC

raccdb1_1

dancer srvpool: backup Oracle GI | HUB

Oracle RAC

raccdb1_2


Oracle RAC

raccdb1_2

comet srvpool: frontoffice Oracle GI | HUB

Oracle RAC

raccdb1_1

1/16/14

53


Policy-Management with Oracle RAC 12c Using Policy Sets – changing policies on the push of a button is fast …






raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1


dancer srvpool: frontoffice dasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB




raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1

[GRID]> date; crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=DayTime"; date Mon Sep 16 19:26:42 PDT 2013 CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded Mon Sep 16 19:26:43 PDT 2013 è Time to execute: 1 second!


Policy-Management with Oracle RAC 12c What-If evaluation of policy changes

[RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend Stage Group 1: ------------------------------------------------------------------------------- Stage Number Required Action ------------------------------------------------------------------------------- 1 Y Server 'comet' will be moved from pools [ora.frontoffice] to pools [ora.backup] Y Server 'dancer' will be moved from pools [ora.frontoffice] to pools [ora.backup] Y Resource 'ora.raccdb1.crmsvc.svc' (1/1) will be in state [OFFLINE] Y Resource 'ora.raccdb1.crmsvc.svc' (2/1) will be in state [OFFLINE] 2 Y Resource 'ora.raccdb1.backup.svc' (1/1) will be in state [ONLINE|INTERMEDIATE] on server[comet] Y Resource 'ora.raccdb1.backup.svc' (2/1) will be in state [ONLINE|INTERMEDIATE] on server [dancer]

vixen srvpool: frontoffice comet srvpool: frontoffice





raccdb1

raccdb1_3

raccdb1_4 raccdb1_2

raccdb1_1

1/16/14

54


What-If with Oracle RAC 12c What-If evaluation of policy changes – in various levels

[RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend -admin -l 'serverpools' NAME = Free ACTIVE_SERVERS = NAME = Generic ACTIVE_SERVERS = NAME = ora.backoffice ACTIVE_SERVERS = vixen NAME = ora.backup ACTIVE_SERVERS = comet dancer NAME = ora.frontoffice ACTIVE_SERVERS = dasher

[RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend -admin -l 'resources' -------------------------------------------------------------------------------- Name Target State Server Effect -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.mgmtdb 1 ONLINE ONLINE dasher ora.raccdb1.backup.svc 1 ONLINE ONLINE comet Started 2 ONLINE ONLINE dancer Started ora.raccdb1.crmsvc.svc 1 ONLINE OFFLINE Stopped 2 ONLINE OFFLINE Stopped 3 ONLINE ONLINE dasher ora.raccdb1.db 1 ONLINE ONLINE comet 2 ONLINE ONLINE dancer 3 ONLINE ONLINE vixen 4 ONLINE ONLINE dasher ora.raccdb1.hrsvc.svc 1 ONLINE ONLINE vixen 2 ONLINE OFFLINE --------------------------------------------------------------------------------


Policy-Management with Oracle RAC 12c Information on each step on the way

Before After

[GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=NightTime“ CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'comet' succeeded

[GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=DayTime" CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded

1/16/14

55



Oracle database with RAC 12c HA BP TCOUG

Technology

Transcript of Oracle database with RAC 12c HA BP TCOUG