Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Markus Michalewicz Director of Product Management Oracle Real Application Clusters July 15, 2014 @OracleRACpm http://www.linkedin.com/in/markusmichalewicz http://www.slideshare.net/MarkusMichalewicz

http://www.linkedin.com/in/markusmichalewicz

http://www.slideshare.net/MarkusMichalewicz

Copyright © 2014 Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

Introduction & Overview

Bronze-Level HA

Silver-Level HA

Gold & Platinum-Level HA

Oracle RAC & Virtualization

Additional Information

5

6


Eliminate risk of downtime and data loss

High Availability (HA) Business Challenges

Reduce cost and increase return on investment


A simplified Two-Step Implementation Approach

Standardize

Set of reference

HA architectures

Reduce costs

Consolidate


Oracle Maximum Availability Architecture (MAA)

Edition-based Redefinition, Online Redefinition, DG / GG – Minimal downtime upgrades & migrations

Active Data Guard – Data Protection, DR – Query Offload

GoldenGate – Active-active replication – Heterogeneous

Active Replica

RMAN, Oracle Secure Backup – Backup to disk, tape or cloud

Application Continuity – Application HA

Global Data Services – Service Failover / Load Balancing

Enterprise Manager Cloud Control

– Coordinated Site Failover

RAC – Scalability – Server HA

Flashback – Human error

correction

Production Site

ASM – Storage management

Database – Core Technology


Oracle Database 12c MAA Four Standard Reference Architectures

GOLD Comprehensive HA and Disaster Protection Zero or near-zero data loss

SILVER High Availability (HA) for Recoverable Local Outages Data protected as of last backup

BRONZE Single Instance Database, Basic Service Restart Data protected as of last backup

PLATINUM Zero Outage for Platinum Ready Applications Zero data loss


BRONZE


SILVER


GOLD


PLATINUM


Standardize on Clusters Clusters increase scalability and HA regardless of DB types used later

High Availability

Scal

abili

ty

License Costs


Standardize on Oracle RAC

High Availability

Scal

abili

ty

For all the flexibility you need


Standardize on Oracle RAC and Oracle Multitenant

High Availability

Scal

abili

ty

The new standard for consolidation

15

Commonwealth Bank

The Commonwealth Bank (CBA) is one of Australia’s leading providers of

integrated financial services including retail, business and institutional banking,

funds management, superannuation, insurance, investment and broking services.

The Bank is one of the largest listed companies on the Australian Stock Exchange.

16

Introduction

In 2007 CBA set out to create an Oracle database shared service offering for the bank

• The offering has been highly successful by several measures

• Oracle-as-a-Service has continued to be developed through several iterations

Oracle as a Service (OaaS) v1 – went live May 2008

17

Host many Oracle database applications

on a cluster of hardware

Processor consolidation

• Run each server hotter

• Take advantage of complimentary workload peaks.

Higher Availability

• Load balancing

• HA failover for component failure

• Standby DR

• Most apps do not implement these features – too expensive

Cost Reduction

Better Service

• Full time experts

• Always on-call

Reduced Risk

• Whole environment is managed

• Operated as a “business”

Oracle as a Service In A Nutshell

18

Reduce Risk, Improve Time to Market

For new Projects:

• Remove a phase from the project – infrastructure already in place

• Remove reliance on expensive/scarce SME resources for design and build

• No longer need to manage risk associated with procurement and build

• Time to instantiate a new Production quality environment: 3 months -> 2 minutes.

Example: New ISV Application introduced into our Online Share Trading platform

• Required to test performance under the workload & data volume conditions projected in 2 years time.

Dedicated Infrastructure OaaS

Implementation Time 3-4 months few hours

$ Cost to Project Several hundred thousand < $10K

On Project Completion Under-utilized asset remains Environment turned-off


Program Agenda

1

2

3

4


Bronze-Level HA

Silver-Level HA




5

6


BRONZE

• Minimize the cost of HA

– Use HA features included with Oracle Database

– Utilize backups to protect against media failures

– Secure offsite tape storage (in the cloud) for DR

• Optionally

– Consolidate with Oracle Multitenant

– Use VMs for dedicated resources

– Self-Service provisioning with

Enterprise Manager Database as a Service

Bronze Level High Availability


BRONZE

High Availability at the Bronze Tier Features Integrated with the Oracle Database

Physical Data Protection

Data Corruption Protection

& Oracle ASM (Mirroring)

Human Error Correction

Flashback Technologies: Drop, Query, Transaction, Table, and Database

Planned Maintenance

Online Redefinition

Online Reorganization

Edition Based Redefinition

Online File Move

Online Patching

Fast Backup & Recovery

Recovery Manager (RMAN)

Fast Recovery Area

Oracle Secure Backup


Oracle Database Inherent Protection Data Corruption Protection

• Oracle Data blocks have a well-defined structure • Block header is kept consistent with payload • Enables validation of both physical and logical intra-block consistency • Oracle ensures block validity is maintained as it traverses I/O path

• Extensive corruption checks: • DB_BLOCK_CHECKSUM: http://docs.oracle.com/cd/E11882_01/server.112/e40402/initparams049.htm • DB_BLOCK_CHECKING: http://docs.oracle.com/cd/E18283_01/server.112/e17110/initparams046.htm • For more information see My Oracle Support (MOS) note 1302539.1

http://docs.oracle.com/cd/E11882_01/server.112/e40402/initparams049.htm

http://docs.oracle.com/cd/E18283_01/server.112/e17110/initparams046.htm


Oracle Automatic Storage Management (ASM) 3rd Party FS Application

Automatic Storage Management

ASM Cluster & Single Node File System (ACFS)

Database

ACFS Snapshot

ASM Disk Group

DB Datafiles, OCR and Voting Files Oracle Binaries 3rd Party File Systems

Dynamic Volume Manager

ASM Instance Managing Oracle DB Files

• ASM supports ALL data – database files, file systems, Clusterware files (OCR, Voting Disk) • Built-in mirroring protects from disk failures • Auto-repair of corrupt blocks using a valid mirror copy


Small ASM Features – Big Difference

Failure Group repair time

Similar to existing disk repair time

Default is 24 hours

Disk group attribute: failgroup_repair_time

Disk Resync

Accepts power limit

Range 1 (least resources) to 1024 (most resources)

Checkpoints: failed operations restart at the last checkpoint

Fast Disk Replacement

Efficient, atomic operation

No disk group reorganization required

Data Scrubbing

Check for logical consistency (and repair from mirror)

during rebalance

areas on demand

Protects seldom-accessed data


Fine-grained Table Recovery From Backup using a simple RMAN command

1 0 1

0 1 0 1 0

1 1 1 0 1 0 0 1

1 0 1 1 0 1 1 1 1 0

0 0 1 0 1 0 0 0 0

1 0 0 1 1 0 1

1 0 1 0

0 1 RMAN Backups

• One simple RECOVER TABLE command to • recover one or more tables • from an RMAN backup (most recent or older)

• Eliminates time and complexity associated with manual restore, recover & export

• Enables fine-grained point-in-time recovery of • individual tables • instead of the contents of the entire tablespace.


Cross-Platform Backup & Restore Simplified Platform Migration

1 1 1 0 1

1 0 1 1 0 1 1 1 1 0 1

0 0 1 0 1 0 0 0 0 1

1 0 0 1

1 1 1 0 1

1 0 1 1 0

1 1 1 1 0 1

0 0

1 0 1 0

0 0

0 1

1 0 0

1

Source Database (AIX)

Backup to Disk/Tape

(data files, optional endian conversion, metadata export)

Restore Backup (optional endian

conversion, metadata import)

Destination Database (Solaris)

• Simplified procedure for platform migration


Program Agenda

1

2

3

4


Bronze-Level HA

Silver-Level HA




5

6


Program Agenda

A

B

C

Silver-Level HA

Free-of-Charge Cluster Features

Stabilize your Cluster

Oracle RAC One Node as a Standard


SILVER

Silver Level High Availability

• Active-passive with Oracle RAC One Node

• Automatic failover

• Fast restart on a second node

• Real-time failover & scalability with Oracle RAC

• All nodes active at all times

• Zero downtime rolling maintenance

• Hardware and OS maintenance

• For qualified Oracle Database patches


Standardize on Clusters Clusters increase scalability and HA regardless of DB types used later

High Availability

Scal

abili

ty

License Costs


Oracle ASM – Overview Oracle Database 11.2 or earlier configuration

ASM Cluster Pool of Storage

Node2

Disk Group B Disk Group A

Node3 Node5 Node4

Shared Disk

Groups

Wide File Striping

One to One

Mapping of ASM

Instances to

Servers

Node1

ASM Instance

Database Instance

ASM Disk

RAC Cluster

DBA DBA DBB DBB DBC DBB

ASM ASM ASM ASM ASM


Oracle Automatic Storage Management (ASM) Oracle ASM 12c Standard Deployment


Disk Group B Disk Group A Shared Disk

Groups

Wide File Striping

One to One

Mapping of ASM

Instances to

Servers

ASM Instance

Database Instance

ASM Disk

RAC Cluster

Node4 Node3 Node2 Node1 Node5 ASM ASM ASM ASM ASM

ASM Instance

Database Instance

DBA DBA DBC DBB DBB DBB


Introducing Oracle Flex ASM Removal of One to One Mapping and HA



Databases share

ASM instances

ASM Instance

Database Instance

ASM Disk

RAC Cluster

Node5 Node4 Node3 Node2 Node1

Node1

runs as

ASM

Client to

Node2

Node1

runs as

ASM

Client to

Node4

Node2

runs as

ASM

Client to

Node3

ASM ASM ASM

ASM Instance

DBA DBA DBC DBB DBB DBB


ACFS – The ASM based Cluster File System ACFS is part of Oracle GI, hosts all data and is fully integrated



Databases share

ASM instances ASM Instance

Database Instance

ASM Disk

Cluster

Node5 Node4 Node3 Node2 Node1 ASM ASM ASM

SI

DB

RAC

DBA

RAC

DBA

RAC

DBA

Applications use

ACFS for all data

ACFS ACFS ACFS ACFS ACFS


Program Agenda

A

B

C

Silver-Level HA





Stabilize your Cluster - Reminder: Network Configuration

vixen comet

racdb1_3 racdb1_4

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

• Size Interconnect for aggregated throughput

– Foresee approx. 1Gb/sec per 32 cores

• 10GbE or IB are recommended for database consolidation and large SMP nodes (use HA)

• Bonded 1GbE with load balancing are most common and supported

– Use same type NICs for LB and failover

– Use different subnets (especially using HAIPs)

• Check for IP & NIC send and receive buffer size

• Use Jumbo frames wherever possible

• Interconnect should be stress tested usign Iperf or netperf


Stabilize Your Cluster – Part 1 Avoid number of sessions related resource contention

vixen comet

racdb1_3 racdb1_4

Connection Pool

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

• Control the number of concurrent sessions • Foreground processes are in time-share class • Scheduling delays on high context switch rates on busy systems may increase the variation in the cluster traffic times • More processes imply higher memory utilization and higher risk of paging

•How to control concurrent sessions: • Use connection pooling • Avoid connection storms (pool and process limits )

• Ensure that load is well-balanced over nodes


Stabilize Your Cluster – Part 2 Optimize memory locally

• Avoid memory pressure!

– Paging and Swapping activity on one node affects performance on all nodes

– Severe Paging and Swapping activity on one node can cause instance evictions

• #1 cause for service disruptions in clusters

• Use Memory Guard

– Oracle Quality of Service Management (QoS) feature – available in monitoring only mode

– Prevents new connections from coming in to a server that is already under memory pressure

vixen comet

racdb1_3 racdb1_4

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC


Stabilize Your Cluster – Part 3 Use Solid State Disks to host swap space

• Use Solid State Disks (SSDs) to host swap space in order to increase node availability

– Memory pressure can cause node evictions.

– Preventing memory pressure is the solution.

– If prevention is not successful and swapping is performed by the Operating System (OS),

• hosting the swap space can mitigate the impact that extensive swapping can have on cluster operations on the on the affected server(s).

• More information:

– My Oracle Support Note Doc ID: 1671605.1 – “Use Solid State Disks to host swap space in order to increase node availability”

vixen comet

racdb1_3 racdb1_4

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

Swapping


Stabilize Your Cluster – Part 4 Configure Huge Pages for Oracle RAC

• Use Huge pages for SGA (Linux)

– Dramatic reduction in memory for page tables

– SGA pages pinned in memory

• More information:

– My Oracle Support note 361323.1 – HugePages on Linux: What It Is... and What It Is Not...

– My Oracle Support note 401749.1 – Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration

• Engineered systems provide templates for pre-configuration of huge pages for the SGA


Program Agenda

A

B

C

Silver-Level HA





Oracle RAC One Node Oracle RAC One Node 12c provides:

1. Best in-class single instance Oracle Database availability

2. Better database consolidation

3. Better database virtualization

Using

• A standardized and improved deployment and management

• A familiar and matured HA stack

• Oracle Grid Infrastructure


Oracle RAC One Node Basic Features • Oracle RAC One Node is

– an Oracle RAC enabled database

– with only one instance running under normal operation

• In case of a server or instance failure, the one running instance is started on another server in the cluster (failover)

• Online Database Relocation (Oracle RAC One Node specific feature) minimizes downtime during scheduled maintenance operations.


Oracle RAC One Node vs. Oracle RAC “Always On” vs. “Always Running”

Oracle RAC One Node: one instance

per database running at a time

Oracle RAC: multiple instances

per database running concurrently


Online Database Relocation for better HA Oracle RAC One Node-specific functionality that makes the difference

Node3 Node2

Oracle (Flex) ASM based pool of shared storage

Node1

Public Network

DBB DBC

Oracle Grid Infrastructure

DBE DBD

DBA

DBA Online DB Relocation

•Online Database Relocation minimizes downtime during (scheduled) maintenance operations such as:

•Patching

•Resource re-allocation due to demand

http://www.oracle.com/goto/racone


Standardize on Oracle RAC (One Node) To simplify Oracle RAC testing

vixen comet

RON RON_2

Online upgrade

RON_1

AWR ---

ASH

AWR ---

ASH

Decide

Oracle GI

Oracle RAC

Oracle GI

Oracle RAC

Oracle RAC testing made simple:

1. Standardize on Oracle RAC (One Node)

2. Run tests on Oracle RAC One Node instance

• RAC One Node behaves basically like a single instance

3. Online Upgrade to Oracle RAC for testing only

4. Run tests on Oracle RAC

5. Decide whether the application provides service goals when load balanced over multiple active instances

• Identify bottlenecks and tuning actions as required


Standardize on Oracle RAC (One Node) The simplest way of converting to RAC: unplug/plug

Node2

CDB Instance 2

Node1

CDB Instance 1

CDB


Oracle RAC Oracle RAC 12c provides:

1. Better Business Continuity and High Availability (HA)

2. Agility and Scalability

3. Cost-effective Workload Management

Using

• A standardized and improved deployment and management

• A familiar and matured HA stack


Program Agenda

1

2

3

4


Bronze-Level HA

Silver-Level HA




5

6


Oracle Database 12c MAA Four Standard Reference Architectures

GOLD Comprehensive HA and Disaster Protection Zero or near-zero data loss

SILVER High Availability (HA) for Recoverable Local Outages Data protected as of last backup

BRONZE Single Instance Database, Basic Service Restart Data protected as of last backup

PLATINUM Zero Outage for Platinum Ready Applications Zero data loss

✔ ✔


GOLD


PLATINUM


Program Agenda

1

2

3

4


Bronze-Level HA

Silver-Level HA




5

6


This is Oracle RAC

Storage Management

1. Oracle ASM

2. Oracle ACFS

Quality of Service

1. Quality of Service Management (QoS)

2. Application Continuity

High Availability and Scalability

1. Oracle Clusterware

2. Oracle RAC (One Node)

Consolidation

1. Oracle RAC with Oracle Multitenant

Core


Optimizing the use of DBs in virtual environments

Bare-Metal Server

DOM-0 Hypervisor

Guest

Dom-0

Guest

Bare-Metal Server

DOM-0 Hypervisor

Dom-0

Guest Guest

DBC1 DBA1

+1

DBC2 DBA2

+1

Oracle RAC provides additional benefit in any virtual environment

Core


Optimizing the use of DBs in virtual environments

Guest Guest Guest Guest

DBC1 DBA1

+1

DBC2 DBA2

+1

Core


Providing an alternative to virtual environments

Node1 Node2 NodeN Node..

DBC1 DBA1 DBC2 DBA2

+1 +1

Core


Program Agenda

1

2

3

4


Bronze-Level HA

Silver-Level HA




5

6


Tools for a Lasting HA Experience

dasher dancer

vixen comet

LOGGERD

sysmond

sysmond

Oracle GI Oracle GI

Oracle GI Oracle GI

sysmond

sysmond

sysmond

LOGGERD

Cluster Health Monitor (CHM) – CHM/OS

1. First failure diagnostics: Collects as much system metrics and data as feasible

2. Collects system metrics on all nodes in the cluster synchronized

3. Persistent storage of all collected data in a database

4. Integrated with Oracle Grid Infrastructure starting with 11.2.0.2 (12c: uses GIMR)

5. Robust collection in situations of resource starvation

6. Data dump or visualization


Tools for a Lasting HA Experience

Guest Guest Guest

DBC1 DBA1 DB DBA2

• Trace File Analyzer (TFA) – Improved comprehensive

first failure diagnostics collection

– Efficient collection, packaging and transfer of data for customers

– Collect for all relevant components (OS, Grid Infra., ASM, RDBMS)

– One command to collect all information

– More information: MOS note ID 1513912.1

• OraChk (formerly RACcheck)

– RAC Configuration Audit Tool

– For details see MOS note ID 1268927.1

– Checks the Oracle Database:

– Standalone Database

– Grid Infrastructure & RAC

– Maximum Availability Architecture (MAA) Validation


More Information: www.oracle.com/goto/rac

• Presentations: •Oracle RAC 12c Collaborate Best Practices

•http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-collaborate-best-practices-ioug-2014-version

•Oracle RAC 12c Practical Performance Management and Tuning •http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-practical-performance-management-and-tuningoow13con8825

•The Oracle RAC Family of Solutions •http://www.slideshare.net/MarkusMichalewicz/the-oracle-real-application-clusters-rac-family-of-solutions

• White Papers: •Oracle RAC 12c Best Practices

•http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-best-practices

http://www.oracle.com/goto/rac

http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-collaborate-best-practices-ioug-2014-version

















http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-practical-performance-management-and-tuningoow13con8825















http://www.slideshare.net/MarkusMichalewicz/the-oracle-real-application-clusters-rac-family-of-solutions

















http://www.slideshare.net/MarkusMichalewicz/oracle-rac-12c-best-practices










• Database and Middleware streams available

• Videos

– Feature different groups and presenters

– Cover a broad range of topics and products

• Oracle RAC PM / Dev contributions so far:

– Oracle Flex Cluster: Optimized Resource Management for the Cloud - Ian Cookson

– Oracle Grid Infrastructure 12c Bundled Agents – Shankar Iyer

– ACFS Product Overview and Use Cases - Ara Shakian

– The Oracle Real Application Clusters (RAC) Family of Solutions - A User Guide – Markus M.

– Next Generation Oracle Automatic Storage Management - Jim Williams

– Implementing DBaaS with Oracle RAC 12c and Quality of Service Management - Mark Scardina

– Oracle RAC Practical Performance Management and Tuning – Markus Michalewicz

Oracle Learning Streams: http://education.oracle.com/streams/

http://education.oracle.com/streams/


• Presentation: Oracle RAC on Extended Distance Clusters – http://www.slideshare.net/MarkusMichalewicz/oracle-rac-on-extended-distance-clusters

• WP: Oracle Real Application Clusters on Extended Distance Clusters – http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf

• WP: Using standard NFS to support a third voting file for extended cluster configurations

– http://www.oracle.com/technetwork/database/database-technologies/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf

• Presentation: Oracle Clusterware Node Management and Voting Disks – http://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voting-disks

More Information: www.oracle.com/goto/rac

http://www.slideshare.net/MarkusMichalewicz/oracle-rac-on-extended-distance-clusters











http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf



http://www.oracle.com/technetwork/database/database-technologies/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf









http://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voting-disks













http://www.oracle.com/goto/rac

Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices

Art & Photos

Transcript of Oracle Database 12c with Real Application Clusters (RAC) High Availability (HA) Best Practices