1 © 2005 Julian Dyke

A Rough Guide to RAC

Julian Dyke

Independent Consultant

Web Version

juliandyke.com

2

© 2005 Julian Dyke

juliandyke.com

Agenda

Introduction Availability Scalability Manageability Total Cost of Ownership Conclusion

3

© 2005 Julian Dyke

juliandyke.com

Introduction

4

© 2005 Julian Dyke

juliandyke.com

Some RAC Terminology

RAC

LMS

LCK

LMD

FAN

LMON

VIP

ONSDIAG

AST

BAST

GRD

GCS

GES

TAF

OCFS

OCFS2

OCR

CRS

FCF

SRVCTLOCRDUMP

VIPCA

GSDLKDEBUG

ASM

PI

CRS_STAT

CRSDOCSSD

CSS

OCRCONFIG

OCRCHECK

CRSCTL

EVMD

OIFCFG

CLUVFY

5

© 2005 Julian Dyke

juliandyke.com

What is RAC? Multiple instances running on separate servers (nodes)

Single database on shared storage accessible to all nodes

Instances exchange information over an interconnect network

Node 1

Instance 1

Node 2

Instance 2Interconnect

Shared Storage

LocalDisk

LocalDisk

6

© 2005 Julian Dyke

juliandyke.com

Instances versus Databases A RAC cluster includes

one database one or more instances

A database is a set of files Located on shared storage Contains all persistent resources

An instance is a set of memory structures and processes Contain all temporal resources Can be started and stopped independently

7

© 2005 Julian Dyke

juliandyke.com

Instances versus Databases

Instance 1

Node 1

Instance 2

Node 2

Instance 3

Node 3

Instance 4

Node 4

Storage Network

PublicNetworkPrivate Network

(Interconnect)

Database

8

© 2005 Julian Dyke

juliandyke.com

What is a RAC Database? Located on shared storage accessible by all instances

Includes Control Files Data Files Online Redo Logs Server Parameter File

May optionally include Archived Redo Logs Backups Flashback Logs (Oracle 10.1 and above) Change Tracking Writer files (Oracle 10.1 and above)

9

© 2005 Julian Dyke

juliandyke.com

What is a RAC Database? Contents similar to single instance database except

One redo thread per instance

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 51200K,GROUP 4 SIZE 51200K;

ALTER DATABASE ENABLE PUBLIC THREAD 2;

If using Automatic Undo Management also require one UNDO tablespace per instance

CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 25600K AUTOEXTEND ON MAXSIZE UNLIMITED EXTENT MANAGEMENT LOCAL;

Additional dynamic performance views (V$, GV$ but not X$) created by $ORACLE_HOME/rdbms/admin/catclust.sql

10

© 2005 Julian Dyke

juliandyke.com

What is the Interconnect? Instances communicate with each other over the interconnect

(network)

Information transferred between instances includes data blocks locks SCNs

Typically 1GB Ethernet UDP protocol Often teamed in pairs to avoid SPOFs

Can also use Infiniband Fewer levels in stack

Other proprietary protocols are available

11

© 2005 Julian Dyke

juliandyke.com

Why Use Shared Storage? Mandatory for

Database files Control files Online redo logs Server Parameter file (if used)

Optional for Archived redo logs (recommended) Executables (Binaries)

Password files Parameter files Network configuration files

Administrative directories Alert Log Dump Files

12

© 2005 Julian Dyke

juliandyke.com

What Shared Storage is Supported? Oracle supplied options

Oracle Cluster File System (OCFS) Version 1

Windows and Linux Supports database and archived redo logs No executables

Version 2 - August 2005 Linux, Windows and Solaris As OCFS1 plus executables

Automatic Storage Management (ASM) Oracle 10.1 and above More transparent in Oracle 10.2 and above

Both require underlying SAN or NAS Do not require LVM

13

© 2005 Julian Dyke

juliandyke.com

What Shared Storage is Supported? Can use (continued)

Network Attached Storage NFS-based Potentially lower cost - no fibre channel required Easy to administer

Raw devices Difficult to administer Cannot be used with archived redo logs

Third-party Cluster File System Still a popular choice with many sites

Others (not supported) Firewire - maximum two nodes - recommended in 10g NBD - Network Block Devices - Solaris and Linux NFS - not supported, but might still work

14

© 2005 Julian Dyke

juliandyke.com

What is a Shared Oracle Home? Can install multiple copies of Oracle executables on local

disks on each node Can also install Shared Oracle Home

single copy of Oracle executables on shared storage

Oracle 9.2 Only Oracle database software

Oracle 10.1 Cluster Ready Services (CRS) Oracle database software + ASM

Oracle 10.2 Oracle Clusterware (CRS) ASM Oracle database software

15

© 2005 Julian Dyke

juliandyke.com

Internal Structures and Services Global Resource Directory (GRD)

Records current state and owner of each resource Contains convert and write queues Distributed across all instances in cluster

Global Cache Services (GCS) Implements cache coherency for database Coordinates access to database blocks for instances Maintains GRD

Global Enqueue Services (GES) Controls access to other resources (locks) including

library cache dictionary cache

16

© 2005 Julian Dyke

juliandyke.com

Background Processes Each RAC instance has set of standard background

processes e.g. PMON SMON LGWR DBWn ARCn

RAC instances use additional background processes to support GCS and GES including LMON LCK0 LMDn LMSn DIAG

17

© 2005 Julian Dyke

juliandyke.com

Portability Most single-instance applications should port to RAC Some exceptions

Application must scale well on single instance Can be difficult to evaluate

Some features do not work e.g. DBMS_ALERT DBMS_PIPE

External inputs/outputs may need modification Flat files etc

Some RAC features require additional coding TAF

Code may need upgrading to use RAC functionality e.g. FCF requires JDBC Implicit Connection Cache

18

© 2005 Julian Dyke

juliandyke.com

Why Do Users Deploy RAC? Users may deploy RAC to achieve

Increasing availability

Increasing scalability

Improving maintainability

Reduction in total cost of ownership

19

© 2005 Julian Dyke

juliandyke.com

Why Do DBAs Deploy RAC? DBAs may want to deploy RAC because:

Realistic next step for experienced Oracle DBAs

Intellectual challenge

Job protection - ties organisation to Oracle technology

Possible improved earnings

It looks good on their CV

20

© 2005 Julian Dyke

juliandyke.com

Availability

21

© 2005 Julian Dyke

juliandyke.com

What is Failover? If one node or instance fails

Node detecting failure will Read redo log of failed instance from last checkpoint Apply redo to datafiles including undo segments (roll

forward) Rollback uncommitted transactions

Cluster is frozen during part of this process

Instance 1

Node 1

Instance 2

Node 2

Interconnect

22

© 2005 Julian Dyke

juliandyke.com

What are Database Services? Database Services are logical groups of sessions

Can be configured using DBCA Enterprise Manager (10.2 and above)

Can also be configured using SRVCTL (Oracle Cluster Registry only) SQL*Plus (Data Dictionary only) Text editor (Network Configuration)

In Oracle 10.1 and above, each service has Preferred Nodes (used by default) Available Nodes (used if preferred node fails)

23

© 2005 Julian Dyke

juliandyke.com

What are Database Services? Can be used with Resource Manager to control resource

usage e.g. CPU Parallel execution

Can be used for monitoring V$SERVICE_STATS

Can be used for diagnostics DBMS_MONITOR

trace statistics

24

© 2005 Julian Dyke

juliandyke.com

What is Oracle Clusterware? Introduced in Oracle 10.1 (Cluster Ready Services - CRS) Renamed in Oracle 10.2 to Oracle Clusterware Cluster Manager providing

Node membership services Global resource management High availability functions

On Linux Configured in /etc/inittab Implemented using three daemons

CRS - Cluster Ready Service CSS - Cluster Synchronization Service EVM - Event Manager

In Oracle 10.2 includes High Availability framework Allows non-Oracle applications to be managed

25

© 2005 Julian Dyke

juliandyke.com

What is the OCR? Oracle Cluster Registry (OCR)

Configuration information for Oracle Clusterware / CRS

Introduced in Oracle 10.1 Replaced Server Management (SRVM) disk/file

Similar to Windows Registry

Located on shared storage

In Oracle 10.2 and above can be mirrored Maximum two copies

26

© 2005 Julian Dyke

juliandyke.com

What is the OCR? Defines cluster resources including:

Databases Instances

RDBMS ASM

Services Node Applications

VIP ONS GSD

Listener Process

27

© 2005 Julian Dyke

juliandyke.com

What is a Voting Disk? Known as Quorum Disk / File in Oracle 9i

Located on shared storage accessible to all instances

Used to determine RAC instance membership

In the event of node failure voting disk is used to determine which instance takes control of cluster Avoids split brain

In Oracle 10.2 and above can be mirrored Odd number of copies (1, 3, 5 etc)

28

© 2005 Julian Dyke

juliandyke.com

What is VIP? Node application introduced in Oracle 10.1

Allows Virtual IP address to be defined for each node

All applications connect using Virtual IP addresses

If node fails Virtual IP address is automatically relocated to another node

Only applies to newly connecting sessions

29

© 2005 Julian Dyke

juliandyke.com

What is TAF? TAF is Transparent Application Failover

Sessions connected to a failed instance will be terminated Uncommitted transactions will be rolled back

Sessions can be reconnected to another instance automatically if using TAF Can optionally re-execute in-progress SELECT statements

Statement re-executed with same SCN Fetches resume at point of failure

Session state is lost including Session parameters Package variables Class and ADT instantiations

30

© 2005 Julian Dyke

juliandyke.com

What is TAF? TAF is Transparent Application Failover

Requires additional coding in client

Requires configuration in TNSNAMES.ORA

RAC_FAILOVER = (DESCRIPTION =

(ADDRESS_LIST =(FAILOVER = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))

)(CONNECT_DATA =

(SERVICE_NAME = RAC)(SERVER = DEDICATED)(FAILOVER_MODE =(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5))

))

31

© 2005 Julian Dyke

juliandyke.com

What is FAN? Fast Application Notification (FAN)

Introduced in Oracle 10.1

Method by which applications can be informed of changes in cluster status Handle node failures Workload balancing

Applications must connect using services

Can be notified using Server side callouts Fast Connection Failover (FCF) ONS API

32

© 2005 Julian Dyke

juliandyke.com

What is ONS? Oracle Notification Service (ONS)

Introduced in Oracle 10.1

Allows out-of-band messages to be sent to Nodes in cluster Middle-tier application servers Clients

Underlying mechanism for Fast Application Notification (FAN)

33

© 2005 Julian Dyke

juliandyke.com

Does RAC Increase Availability? Depends on definition of availability

May achieve less unplanned downtime May have more time to respond to failures

Instance failover means any node can fail without total loss of service

Must provide have overcapacity in cluster to survive failover Additional Oracle and RAC licenses Load can be distributed over all running nodes Can use Grid to provision additional nodes

34

© 2005 Julian Dyke

juliandyke.com

Does RAC Increase Availability? Can still get data corruptions

Human errors / software errors Only one logical copy of data Only one logical copy of application / Oracle software

Lots of possibility for human errors Power / network cabling / storage configuration

Upgrades and patches are more complex Can upgrade software on subset of nodes If database is affected then still need downtime

35

© 2005 Julian Dyke

juliandyke.com

Scalability

36

© 2005 Julian Dyke

juliandyke.com

What is Scalability? RAC overhead means that linear scalability is difficult to

achieve Global Cache Services (blocks) Global Enqueue Services (locks)

As number of instances increases, probability that instance is a resource master decreases

Scaling factor of 1.8 is considered good

Dependent on application design and implementation

Scaling factor improves with Node affinity Elimination of contention

37

© 2005 Julian Dyke

juliandyke.com

What is Scalability? Scalability is the relationship between increments of

resources and workloads Can be any resource but with RAC normally refers to adding

instances Scalability can be

linear - optimal but rare non-linear - suboptimal but normal

Resource

Wo

rklo

ad

Resource

Wo

rklo

ad

Linear Non-Linear

38

© 2005 Julian Dyke

juliandyke.com

What is Workload Balancing? Balancing of workload across available instances Can have

Client-side connection balancing Server-side connection balancing

Client-side connection balancing Workload distributed randomly across nodes

RAC = (DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))(LOAD_BALANCE = ON)(FAILOVER = ON)(CONNECT_DATA =

(SERVICE_NAME = RAC) (FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)) )

)

39

© 2005 Julian Dyke

juliandyke.com

What is Workload Balancing? Server-side connection balancing Dependent on current workload on each node PMON monitors workload and updates listeners Depends on long or short connections

In Oracle 10.1 Set PREFER_LEAST_LOADED_NODE in listener.ora

OFF for long connections ON for short connections (default)

In Oracle 10.2 Can specify load balancing goal for each service

NONE, SERVICE_TIME or THROUGHPUT Can also specify connection load balancing goal

SHORT or LONG

40

© 2005 Julian Dyke

juliandyke.com

Increasing Scalability If application scales well on a single-instance then it should

scale well on RAC

Eliminate contention Use sequences

Use locally partitioned tables and indexes Attempt to achieve node affinity

Avoid contention for single blocks Distribute rows for hot blocks

Small block size e.g. 2048 or 4096 ALTER TABLE MINIMIZE RECORDS PER BLOCK High PCTFREE / Low PCTUSED Filler columns e.g. CHAR (2000)

41

© 2005 Julian Dyke

juliandyke.com

Increasing Scalability Use Automatic Segment Space Management

Default in Oracle 10.2

Use larger block size for read-only objects Reduce number of GCS messages required

Minimize lock usage Eliminate unnecessary parsing

Increase size of shared pool Bind variables Cursor sharing

Use optimistic locking Eliminate unnecessary SELECT FOR UPDATE statements

42

© 2005 Julian Dyke

juliandyke.com

Manageability

43

© 2005 Julian Dyke

juliandyke.com

Server Parameter File Introduced in Oracle 9.0.1 Must reside on shared storage Shared by all RAC instances Binary (not text) files Parameters can be changed using ALTER SYSTEM Can be backed up using the Recovery Manager (RMAN) Created using

CREATE SPFILE [ = ‘SPFILE_NAME’ ]FROM PFILE [ = ‘PFILE_NAME’ ];

SPFILE = <pathname>

init.ora file on each node must contain SPFILE parameter

44

© 2005 Julian Dyke

juliandyke.com

Parameters RAC uses same parameters as single-instance

Some must be different on each instance Some must be same on each instance

Can be global or local

ALTER SYSTEM SET parameter = value[ SCOPE = MEMORY | SPFILE | BOTH ] [ SID = <sid>]

ALTER SYSTEM RESET parameter = value[ SCOPE = MEMORY | SPFILE | BOTH ] [ SID = <sid>]

[*.]<parameter_name> = <value>[<sid>]<parameter_name> = <value>

Must be set using ALTER SYSTEM statement

45

© 2005 Julian Dyke

juliandyke.com

Parameters Some parameters must be same on each instance including *:

ACTIVE_INSTANCE_COUNT ARCHIVE_LAG_TARGET CLUSTER_DATABASE CONTROL_FILES DB_BLOCK_SIZE DB_DOMAIN DB_FILES DB_NAME DB_RECOVERY_FILE_DEST DB_RECOVERY_FILE_DEST_SIZE DB_UNIQUE_NAME MAX_COMMIT_PROPAGATION_DELAY TRACE_ENABLED UNDO_MANAGEMENT

* Correct for Oracle 10.1

46

© 2005 Julian Dyke

juliandyke.com

Parameters Some parameters, if used, must be different on each instance

including THREAD INSTANCE_NUMBER INSTANCE_NAME UNDO_TABLESPACE ROLLBACK_SEGMENTS

DML_LOCKS must be identical on each instance if set to zero

47

© 2005 Julian Dyke

juliandyke.com

DBCA Can be used to

Create RAC database and instances Create ASM instance Manage ASM instance (10.2) Add RAC instances Create RAC database templates

structure only with data

Create clone RAC database (10.2) Create, Manage and Drop Services Drop instances and database

48

© 2005 Julian Dyke

juliandyke.com

What is SRVCTL? Utility used to manage cluster database Configured in Oracle Cluster Registry (OCR) Controls

Database Instance ASM Listener Node Applications Services

Options include Start / Stop Enable / Disable Add / Delete Show current configuration Show current status

49

© 2005 Julian Dyke

juliandyke.com

SRVCTL - Examples Starting and Stopping a Database

srvctl start database -d RACsrvctl stop database -d RAC

srvctl start instance -d RAC -i RAC1srvctl stop instance -d RAC -i RAC1

Starting and Stopping an Instance

Starting and Stopping a Service

srvctl start service -d RAC -s SERVICE1srvctl stop service -d RAC -s SERVICE1

Starting and Stopping ASM on a specified node

srvctl start asm -n node1srvctl stop asm -n node1

50

© 2005 Julian Dyke

juliandyke.com

Enterprise Manager In Oracle 10.1 and above

Database Control Installed by DBCA Controls single cluster

Grid Control Uses separate repository Oracle 10.2 version available

Requires Oracle 10.1 database

Fully supports RAC in both versions Except

Oracle 10.1 cannot create / delete services Oracle 10.2 better interconnect performance monitoring

51

© 2005 Julian Dyke

juliandyke.com

What is CLUVFY? Introduced in Oracle 10.2

Supplied with Oracle Clusterware Can be downloaded from OTN (Linux and Windows)

Written in Java - requires JRE (supplied)

Also works with 10.1 (specify -10gR1 option)

Checks cluster configuration stages - verifies all steps for specified stage have been

completed components - verifies specified component has been

correctly installed

52

© 2005 Julian Dyke

juliandyke.com

CLUVFY Stages include

-post hwos post check for hardware and operating system

-pre cfs pre-check for CFS setup

-post cfs post-check for CFS setup

-pre crsinst pre-check for Oracle Clusterware installation

-post crsinst post-check for Oracle Clusterware installation

-pre dbinst pre-check for database installation

-pre dbcfg pre-check for database configuration

53

© 2005 Julian Dyke

juliandyke.com

CLUVFY Components include

nodereach Checks reachability between nodes

nodecon Checks node connectivity

cfs Checks CFS integrity

ssa Checks shared storage accessibility

space Checks space availability

sys Checks minimum system requirements

clu Checks cluster integrity

clumgr Checks cluster manager integrity

ocr Checks OCR integrity

crs Checks Oracle Clusterware (CRS) integrity

nodeapp Checks node applications exist

admprv Checks administrative privileges

peer Compares properties with peers

54

© 2005 Julian Dyke

juliandyke.com

CLUVFY For example, to check configuration before installing Oracle

Clusterware on node1 and node2 use:

sh runcluvfy.sh stage -pre crsinst -n node1,node2

Checks: node reachability user equivalence administrative privileges node connectivity shared stored accessibility

If any checks fail append -verbose to display more information

55

© 2005 Julian Dyke

juliandyke.com

Other Utilities Additional RAC utilities and diagnostics include

OCRCONFIG OCRCHECK OCRDUMP CRSCTL CRS_STAT

Additional RAC diagnostics can be obtained using ORADEBUG utility

DUMP option LKDEBUG option

Events

56

© 2005 Julian Dyke

juliandyke.com

Does RAC Improve Manageability? Advantages

Fewer databases to manage Easier to monitor Easier to upgrade Easier to control resource allocation Resources can be shared between applications

Disadvantages Upgrades potentially more complex Downtime may affect more applications Requires more experienced operational staff

Higher cost / harder to replace

57

© 2005 Julian Dyke

juliandyke.com

Total Cost ofOwnership

58

© 2005 Julian Dyke

juliandyke.com

Reduction in TCO? Possible for sites with legacy systems

Mainframes / Minicomputers Applications / Packages

RAC option adds 50% to licence costs except for Users with site licences Standard edition (10.1+, max 4 CPU with ASM)

Retrain existing staff or use dedicated staff

Consolidation may bring economies of scale Monitoring Backups Disaster Recovery

59

© 2005 Julian Dyke

juliandyke.com

Reduction in TCO? Additional resources required

Redundant hardware Nodes Network switches SAN fabric Hardware e.g. fibre channel cards

Reduction in hardware support costs May not require 24 hour support Viable to hold stock of spare components

60

© 2005 Julian Dyke

juliandyke.com

What are the Alternatives to RAC? Data Guard

Physical Standby Introduced in Oracle 7.3.4 Stable, well proven technology Requires redundant hardware Implemented by many sites Can be used with RAC

Logical Standby Introduced in Oracle 9.2 Still not widely adopted

Streams Introduced in Oracle 9.2 Implemented by increasing number of sites

Advanced Replication

61

© 2005 Julian Dyke

juliandyke.com

What are the Alternatives to RAC? Symmetric Multiprocessing (SMP) Systems

Single Point of Failure Simplified configuration Eliminate RAC overhead

Parallel systems For systems with deterministic input Messaging Data Warehouses

Other Clustering Technologies SAN Operating System etc

62

© 2005 Julian Dyke

juliandyke.com

Conclusion Success of RAC deployments dependent on

Application design and implementation Failover requirements IT infrastructure Flexibility and commitment of IT department(s)

Before deploying RAC Investigate and reject alternatives Perform proof of concept

Test application Evaluate benefits and costs Learn RAC concepts and administration

Buy a good book :)

63

© 2005 Julian Dyke

juliandyke.com

Thank you for your interest

For more information and to provide feedback

please contact me

My e-mail address is:info@juliandyke.com

My website address is:

www.juliandyke.com