7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 1/47

Oracle Real Application Clusters (RAC)

When you have very critical systems that require to be online 24x7 then you need a HA solution (High Availability)

Sometimes in large corporations you ill hear the phrase five nines! this phrases means the availability o" 

a system and hat dontime (approx) is alloed! the table belo highlights the uptime a system requiresin order to achieve the five nines

% uptime % Downtime Downtime per year Downtime per week

98 2 7#$ days $ hours 22 minutes

99 % $#&' days % hour 4% minutes

99.8 #2 %7 hours $ minutes 2 minutes

99.9 #% hours 4' minutes % minutes

99.99 #% '2#' minutes % minute

99.999 (five nines) #% '#2' minutes & seconds

*o achieve the five nines your system is only alloed '#2' minutes per year or & seconds per ee+! insome HA designs it may ta+e & seconds to "ailover#

When loo+ing "or a solution you should try and build redundancy into your plan! this is the "irststep to a HA solution#

,ou are trying to eliminate as many Single -oint ." /ailures (S-./0s) as you can ithout increasing thecosts#

HA comes in three1"our "lavors

No-failoer 

*his option usually uses the already builtin redundancy! "ailed dis+s and -S3 can be replaced

online! but i" a maor hardare as to "ail then a system outage is unavoidable! the system illremain don until it is "ixed#

*his solution can be per"ectly acceptable in some environments but at hat price to thebusiness! even in today0s mar+et 5A168 systems cost money hen not running! i am surethat your developers are quite happy to ta+e the day o"" paid hile you "ix the system

Cluster 

*his is the eel in the HA orld! a cluster can be con"igure in a variety o" "avors! "rom minimaldontime hile services are moved to a good nodes! to virtual 9ero dontime#

Hoever a cluster solution does come ith a heavy price tag! hardare! con"iguration andmaintaining a cluster is expensive but i" you business loses vast amounts o" money i" yousystem is don! then its orth it#

Col!failoer 

:any smaller companies use this solution! basically you have a additional server ready to ta+e

over a number o" servers i" one server is "ail# ; have used this technique mysel"! i create anumber o" scripts that can turn a cold standby server into any number o" servers! i" the originalserver is going to have a prolonged outage#

*he problem ith this solution is there is going to be dontime! especially i" it ta+es a long timeto get the standby server up to the same point in time as the "ailed server#*he advantage o" thissolution is that one additional server could cover a number o" servers! even i" it slight underpoered to the original server! as long as it +eeps the service running#

"ot :any applications o""er hotstandby servers! these servers are running along side the live

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 2/47

failoer 

system! data is applied to the hotstandby server periodically to +eep it up to date! thus in a"ailover situation the server is almost ready to go#

*he problem ith this system is costs and manageability! also one server is usually dedicated toone application! thus you may have to have many hotstandby servers#

*he advantage is that dontime is +ept to a minimum! but there ill be some dontime!generally the time it ta+e to get the hotstandby server up to date! "or example applying the lastset o" logs to a database#

Here is a summary table that shos the most command aspects o" cold "ailover versus hot "ailover 

Aspects Col! #ailoer "ot #ailoer

$calaility&numerof no!es

Scalable limited to the capacity o" asingle node

 As nodes can be added on demand! itprovides in"inite scalability# High number o"nodes supported#

'ser interruption

<equired up to a minimal extent# *he

"ailover operation can be scripted orautomated to a certain extent

=ot required! "ailover is automatic

ransparent failoer of applications

=ot -ossible*ransparent application "ailover ill beavailable here sessions can be trans"erredto another node ithout user interruption

oa! *alancin+=ot possible! only one server ill beused

;ncoming load can be balanced beteen bothnodes

'sa+e of resources.nly one server at a time! the otherserver ill be +ept idle

>oth the servers ill be used

#ailoer time:ore than minutes as the othersystem must be cold started

?ess than a minute! typically in a "eseconds#

Clusterin+

 A cluster is a group o" to or more interconnected nodes that provide a service# *he cluster provides ahigh level o" "ault tolerance! i" a node ere to become unavailable ithin the cluster the services aremoved1restored to another or+ing node! thus the end user should never +no that a "ault occurred#

;t is very scalable because additional nodes can be added or ta+en aay (a node may need to bepatched) ithout interrupting the service# *here are no three types o" clustering architecture

$,are!not,in+

ach node ithin the cluster is independent! they share nothing# An example o" this may beeb servers! you a have number o" nodes ithin the cluster supplying the same ebservice# *he content ill be static thus there is no need to share dis+s! etc#

$,are! !iskonly

ach node ill be attached or have access to the same set o" dis+s# *hese dis+s ill containthe data that is required by the service# .ne node ill control the application and the dis+and in the event o" a node "ails@ the other node ill ta+e control o" both the application andthe data# *his means that one node ill have to be on standby setting idle aiting to ta+eover i" required to do so#

 A typical traditional 8eritas luster  and Sun luster  ould "it the bill here#

$,are!  Again all nodes ill be attached or have access to the same set o" dis+s! but this time each

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 3/47

eeryt,in+

node can read1rite to the dis+s concurrently# =ormally there ill be a piece o" so"tare thatcontrols the reading and riting to the dis+s ensuring data integrity# *o achieve this a clusteride "ile system is introduced! so that all nodes vie the "ilesystem identically! the so"tarethen coordinates the sharing and updating o" "iles! records and databases#

.racle <A and ;>: HA:- ould be good examples o" this type o" cluster

Oracle RAC "istory

.racle <eal Application luster as introduced in .racle Bi version# >e"ore .racle <A there as .racleparallel server available#

.racle <A addresses the limitation in .-S by extending ache /usion! and the dynamic loc+mastering# .racle %g <A also comes ith its on integrated clusterare and storage management"rameor+! removing all dependencies o" a thirdparty clusterare product#

RAC Arc,itecture

.racle <eal Application clusters allos multiple instances to access a single database@ the instances illbe running on multiple nodes# ;n a standard .racle con"iguration a database can only be mounted by oneinstance but in a <A environment many instances can access a single database#

.racle0s <A is heavy dependent on an e""icient! high reliable high speed private netor+ called theinterconnect! ma+e sure hen designing a <A system that you get the best that you can a""ord#

*he table belo describes the di""erence o" a standard oracle database (single instance) an a <Aenvironment

Component$in+le nstancenironment

RAC nironment

$/A ;nstance has its on SCA ach instance has its on SCA*ack+roun!processes

;nstance has its on set o"bac+ground processes

ach instance has its on set o" bac+ground processes

Datafiles Accessed by only oneinstance

Shared by all instances (shared storage)

Control #iles Accessed by only oneinstance

Shared by all instances (shared storage)

Online Re!o 6edicated "or rite1read to Only one instance can rite but other instances can

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 4/47

o+file only one instanceread during recovery and archiving# ;" an instance isshutdon! log sitches by other instances can "orce theidle instance redo logs to be archived

Arc,ie! Re!oo+file

6edicated to the instance-rivate to the instance but other instances ill needaccess to all required archive logs during mediarecovery

#las, Recoeryo+

 Accessed by only oneinstance

Shared by all instances (shared storage)

Alert o+ an!race #iles

6edicated to the instance-rivate to each instance! other instances never read orrite to those "iles#

ORAC0"O1

:ultiple instances on thesame server accessingdi""erent databases can usethe same executable "iles

Same as single instance plus can be placed on shared"ile system alloing a common .<A?DH.: "or allinstances in a <A environment#

RAC Components

*he maor components o" a .racle <A system areE

• Shared dis+ system

• .racle lusterare

• luster ;nterconnects

• .racle Fernel omponents

*he belo diagram describes the basic architecture o" the .racle <A environment

Disk arc,itecture

With today0s SA= and =AS dis+ storage systems! sharing storage is "airly easy and is required "or a <Aenvironment! you can use the belo storage setups

• SA= (Storage Area =etor+s) generally using "ibre to connect to the SA=

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 5/47

• =AS ( =etor+ Attached Storage) generally using a netor+ to connect to the =AS using either=/S! ;SS;

• G>.6 direct attached storage! the old traditional ay and still used by many companies as acheap option

 All o" the above solutions can o""er multipathing to reduce S-./s ithin the <A environment! there isno reason not to con"igure multipathing as the cost is cheap hen adding additional paths to the dis+because most o" the expense is paid hen out hen con"iguring the "irst path! so an additional controllercard and netor+1"ibre cables is all that is need#

*he last thing to thin+ about is ho to setup the underlining dis+ structure this is +non as a raid level!there are about %2 di""erent raid levels that ; +no o""! here are the most common ones

rai! 2

($tripin+)

 A number o" dis+s are concatenated together to give the appearance o" one very large dis+#

 Advantages  ;mproved per"ormance

  an reate very large 8olumes

6isadvantages  =ot highly available (i" one dis+ "ails! the volume "ails)

rai! 3(1irrorin+)

 A single dis+ is mirrored by another dis+@ i" one dis+ "ails the system is una""ected as it canuse its mirror#

 Advantages  ;mproved per"ormance  Highly Available (i" one dis+ "ails the mirror ta+es over)

6isadvantages  xpensive (requires double the number o" dis+s)

rai! 4

<aid stands "or <edundant Array o" ;nexpensive 6is+s! the dis+s are striped ith parityacross $ or more dis+s! the parity is used in the event that one o" the dis+s "ails! the data onthe "ailed dis+ is reconstructed by using the parity bit#

 Advantages  ;mproved per"ormance (read only)  =ot expensive

6isadvantages  Slo rite operations (caused by having to create the parity bit)

*here are many other raid levels that can be used ith a particular hardare environment "or example: storage uses the <A;6S! H- storage uses Auto <A;6! so chec+ ith the manu"acture "or the best

solution that ill provide you ith the best per"ormance and resilience#

.nce you have you storage attached to the servers! you have three choices on ho to setup the dis+s

• <a 8olumes normally used "or per"ormance bene"its! hoever they are hard to manage andbac+up

• luster /ilesystem used to hold all the .racle data"iles can be used by indos and ?inux! itsnot used idely

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 6/47

•  Automatic Storage :anagement (AS:) .racle choice o" storage management! its a portable!dedicated and optimi9ed cluster "ilesystem

Oracle Clusterware

.racle lusterare so"tare is designed to run .racle in a cluster mode! it can support you to &4 nodes@

it can even be used ith a vendor cluster li+e Sun luster#

*he lusterare so"tare allos nodes to communicate ith each other and "orms the cluster that ma+esthe nodes or+ as a single logical server# *he so"tare is run by the luster <eady Services (<S) usingthe .racle luster <egistry (.<) that records and maintains the cluster and node membershipin"ormation and the voting dis+ hich acts as a tiebrea+er during communication "ailures# onsistentheartbeat in"ormation travels across the interconnect to the voting dis+ hen the cluster is running#

*he <S has "our components

• .-<.d -rocess :onitor 6aemon

• <Sd <S daemon! the "ailure o" this daemon results in a node being reboot to avoid data

corruption

• .SSd .racle luster Synchroni9ation Service 6aemon (updates the registry)

• 8:d vent 8olume :anager 6aemon

*he .-<.d daemon provides the ;1. "encing "or the .racle cluster@ it uses the hangchec+ timer oratchdog timer "or the cluster integrity# ;t is loc+ed into memory and runs as a realtime processes! "ailureo" this daemon results in the node being rebooted# /encing is used to protect the data! i" a node ere tohave problems "encing presumes the orst and protects the data thus restarts the node in question! itsbetter to be save than sorry#

*he <Sd process manages resources such as starting and stopping the services and "ailover o" the

application resources! it also spans separate processes to manage application resources# <Smanages the .< and stores the current +no state o" the cluster! it requires a public! private and 8;-inter"ace in order to run# .SSd provides synchroni9ation services among nodes! it provides access tothe node membership and enables basic cluster services! including cluster group services and loc+ing!"ailure o" this daemon causes the node to be rebooted to avoid split-brain situations#

*he belo "unctions are covered by the .SSd

• SS provides basic Croup Services Support! it is a distributed group membership system thatallos applications to coordinate activities to archive a common result#

• Croup services use vendor clusterare group services hen it is available#

• ?oc+ services provide the basic clusteride seriali9ation loc+ing "unctions! it uses the /irst ;n!

/irst .ut (/;/.) mechanism to manage loc+ing

• =ode services uses .< to store data and updates the in"ormation during recon"iguration! it alsomanages the .< data hich is static otherise#

*he last component is the vent :anagement ?ogger! hich runs the 8:d process# *he daemonspans a processes called evmlogger  and generates the events hen things happen# *he evmlogger  spans ne children processes on demand and scans the callout directory to invo+e callouts# 6eath o"the 8:d daemon ill not halt the instance and ill be restarted#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 7/47

5uic+ recap

CR$ 5rocess #unctionality #ailure of t,e 5rocess Run A$

O5ROC! - 5rocess 1onitor provides basic cluster integrityservices

=ode <estart root

61! - ent 1ana+ement spans a child process event logger and generates callouts 6aemon automaticallyrestarted! no node restart oracle

OC$$! - Cluster$ync,roni7ation $erices

basic node membership! groupservices! basic loc+ing

=ode <estart oracle

CR$! - Cluster Rea!y$erices

resource monitoring! "ailover andnode recovery

6aemon restartedautomatically! no node restart

root

*he clusterready services (<S) is a ne component in %g <A! its is installed in a separate homedirectory called .<A?D<SDH.:# ;t is a mandatory component but can be used ith a third partycluster (8eritas! Sun luster)! by de"ault it manages the node membership "unctionality along ithmanaging regular <Arelated resources and services

<A uses a membership scheme! thus any node anting to oin the cluster as to become a member#<A can evict any member that it seems as a problem@ its primary concern is protecting the data# ,oucan add and remove nodes "rom the cluster and the membership increases or decrease! hen netor+problems occur membership becomes the deciding "actor on hich part stays as the cluster and hatnodes get evicted! the use o" a voting dis+ is used hich ; ill tal+ about later#

*he resource management "rameor+ manage the resources to the cluster (dis+s! volumes)! thus youcan have only have one resource management "rameor+ per resource# :ultiple "rameor+s are notsupported as it can lead to undesirable a""ects#

*he .racle luster <eady Services (<S) uses the registry to +eep the cluster con"iguration! it shouldreside on a shared storage and accessible to all nodes ithin the cluster# *his shared storage is +non asthe .racle luster <egistry (.<) and its a maor part o" the cluster! it is automatically bac+ed up (every

4 hours) the daemons plus you can manually bac+ it up# *he .SSd uses the .< extensively andrites the changes to the registry#

*he .< +eeps details o" all resources and services@ it stores name and value pairs o" in"ormation suchas resources that are used to manage the resource equivalents by the <S stac+# <esources ith the<S stac+ are components that are managed by <S and have the in"ormation on the good1bad stateand the callout scripts# *he .< is also used to supply bootstrap in"ormation ports! nodes! etc! it is abinary "ile#

*he .< is loaded as cache on each node! each node ill update the cache then only one node isalloed to rite the cache to the .< "ile! the node is called the master# *he nterprise manager alsouses the .< cache@ it should be at least %:> in si9e# *he <S daemon ill update the .< aboutstatus o" the nodes in the cluster during recon"igurations and "ailures#

*he voting dis+ (or quorum dis+) is shared by all nodes ithin the cluster! in"ormation about the cluster isconstantly being ritten to the dis+! this is +non as the heartbeat# ;" "or any reason a node cannotaccess the voting dis+ it is immediately evicted "rom the cluster! this protects the cluster "rom splitbrains(the ;nstance :embership <ecovery algorithm ;:< is used to detect and resolve splitbrains) as thevoting dis+ decides hat part is the really cluster# *he voting dis+ manages the cluster membership andarbitrates the cluster onership during communication "ailures beteen nodes# 8oting is o"ten con"usedith quorum they are similar but distinct! belo details hat each means

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 8/47

6otin+  A vote is usually a "ormal expression o" opinion or ill in response to a proposed decision

uorumis de"ined as the number! usually a maority o" members o" a body! that! hen assembledis legally competent to transact business

*he only vote that counts is the quorum member vote@ the quorum member vote de"ines the cluster# ;" anode or group o" nodes cannot archive a quorum! they should not start any services because they ris+con"licting ith an established quorum#

*he voting dis+ has to reside on shared storage@ it is a small "ile (2:>) that can be accessed by allnodes in the cluster# ;n .racle %g <% you can have only one voting dis+! but in <2 you can have upto $2voting dis+s alloing you to eliminate any S-./0s#

*he original 8irtual ;- in .racle as *ransparent Application /ailover (*A/)! this had limitations! and thishas no been replaced ith cluster VIPs. *he cluster VIPs ill "ailover to or+ing nodes i" a node should"ail! these public ;-s are con"igured in 6=S so that users can access them# *he cluster VIPs are di""erent"rom the cluster interconnect ;- address and are only used to access the database#

*he cluster interconnect is used to synchroni9e the resources o" the <A cluster! and also used totrans"er some data "rom one instance to another# *his interconnect should be private! highly available and"ast ith lo latency! ideally they should be on a minimum private %C> netor+# Whatever hardare youare using the =; should use multipathing (?inux bonding! Solaris ;-:-)# ,ou can use crossovercables in a 5A168 environment but it is not supported in a production environment! also crossovercables limit you to a to node cluster#

Oracle ernel Components

*he +ernel components relate to the bac+ground processes! bu""er cache and shared pool and managingthe resources ithout con"licts and corruptions requires special handling#

;n <A as more than one instance is accessing the resource! the instances require better coordination atthe resource management level# ach node ill have its on set o" bu""ers but ill be able to request andreceive data bloc+s currently held in another instance0s cache# *he management o" data sharing andexchange is done by the Clobal ache Services (CS)#

 All the resources in the cluster group "orm a central repository called the Clobal <esource 6irectory(C<6)! hich is distributed# ach instance masters some set o" resources and together all instances "ormthe C<6# *he resources are equally distributed among the nodes based on their eight# *he C<6 ismanaged by to services called Clobal aches Services (CS) and Clobal nqueue Services (CS)!together they "orm and manage the C<6# When a node leaves the cluster! the C<6 portion o" thatinstance needs to be redistributed to the surviving nodes@ a similar action is per"ormed hen a ne node

 oins#

RAC *ack+roun! 5rocesses

ach node has its on bac+ground processes and memory structures! there are additional processesthan the norm to manage the shared resources! theses additional processes maintain cache coherencyacross the nodes#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 9/47

ache coherency is the technique o" +eeping multiple copies o" a bu""er consistent beteen di""erent.racle instances on di""erent nodes# Clobal cache management ensures that access to a master copy o"a data bloc+ in one bu""er cache is coordinated ith the copy o" the bloc+ in another bu""er cache#

*he sequence o" an operation ould go as belo

%# When instance A needs a bloc+ o" data to modi"y! it reads the boc+ "rom dis+! be"ore reading itmust in"orm the CS (6?:)# CS +eeps trac+ o" the loc+ status o" the data bloc+ by +eeping anexclusive loc+ on it on behal" o" instance A

2# =o instance > ants to modi"y that same data bloc+! it too must in"orm CS! CS ill thenrequest instance A to release the loc+! thus CS ensures that instance > gets the latest versiono" the data bloc+ (including instance A modi"ications) and then exclusively loc+s it on instance >behal"#

$# At any one point in time! only one instance has the current copy o" the bloc+! thus +eeping theintegrity o" the bloc+#

CS maintains data coherency and coordination by +eeping trac+ o" all loc+ status o" each bloc+ that canbe read1ritten to by any nodes in the <A# CS is an in memory database that contains in"ormation

about current loc+s on bloc+s and instances aiting to acquire loc+s# *his is +non as Parallel CacheManagement  (-:)# *he Clobal <esource :anager (C<:) helps to coordinate and communicate theloc+ requests "rom .racle processes beteen instances in the <A# ach instance has a bu""er cache inits SCA! to ensure that each <A instance obtains the bloc+ that it needs to satis"y a query ortransaction# <A uses to processes the CS and CS hich maintain records o" loc+ status o" eachdata "ile and each cached bloc+ using a C<6#

So hat is a resource! it is an identi"iable entity! it basically has a name or a re"erence! it can be a area inmemory! a dis+ "ile or an abstract entity# A resource can be oned or loc+ed in various states (exclusive or shared)# Any shared resource is loc+able and i" it is not shared no access con"lict ill occur#

 A global resource is a resource that is visible to all the nodes ithin the cluster# 6ata bu""er cache bloc+sare the most obvious and most heavily global resource@ transaction enqueue0s and database data

structures are other examples# CS handle data bu""er cache bloc+s and CS handle all the nondatabloc+ resources#

 All caches in the SCA are either global or local! dictionary and bu""er caches are global! large and avapool bu""er caches are local# ache "usion is used to read the data bu""er cache "rom another instanceinstead o" getting the bloc+ "rom dis+! thus cache "usion moves current copies o" data bloc+s beteeninstances (hence hy you need a "ast private netor+)! CS manages the bloc+ trans"ers beteen theinstances#

/inally e get to the processes

Oracle RAC Daemons an! 5rocesses

1$n ?oc+ :anager Serverprocess CS

*his is the cache "usion part and the most active process@ it handles theconsistent copies o" bloc+s that are trans"erred beteen instances# ;t receivesrequests "rom ?:6 to per"orm loc+ requests# ;t rolls bac+ any uncommittedtransactions# *here can be up to ten ?:S processes running and can be starteddynamically i" demand requires it#

*hey manage loc+ manager service requests "or CS resources and send themto a service queue to be handled by the ?:Sn process# ;t also handles global

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 10/47

deadloc+ detection and monitors "or loc+ conversion timeouts#

 As a per"ormance gain you can increase this process priority to ma+e sure -3starvation does not occur

,ou can see the statistics o" this daemon by loo+ing at the vie IJFG:S6-

1ON?oc+ :onitor-rocess CS

*his process manages the CS@ it maintains consistency o" CS memorystructure in case o" process death# ;t is also responsible "or clusterrecon"iguration and loc+s recon"iguration (node oining or leaving)! it chec+s "orinstance deaths and listens "or local messaging#

 A detailed log "ile is created that trac+s any recon"igurations that have happened#

1D?oc+ :anager 6aemon CS

*his manages the enqueue manager service requests "or the CS# ;t alsohandles deadloc+ detention and remote resource requests "rom other instances#

you can see the statistics o" this daemon by loo+ing at the vie IJFG:66-

C2 ?oc+ -rocess CS

:anages instance resource requests and crossinstance call operations "or

shared resources# ;t builds a list o" invalid loc+ elements and validates loc+elements during recovery#

DA/6iagnostic6aemon

*his is a lighteight process@ it uses the 6;AC "rameor+ to monitor the health o" the cluster# ;t captures in"ormation "or later diagnosis in the event o" "ailures# ;till per"orm any necessary recovery i" an operational hang is detected#

RAC A!ministration

 At the time o" <A installation! ma+e use o" that third node! don0t install it ith the original con"iguration!add it a"terards! use this node to remove a node "rom the cluster and also to simulate node "ailures! thisis the only ay to learn! +eep repeating certain situations until you "ully understand ho <A or+s#

;t is recommended that the sp"ile (binary parameter "ile) is shared beteen all nodes ithin the cluster! butit is possible that each instance can have its on sp"ile# *he parameters can be grouped into threecategories#

'ni:ue parameters*hese parameters are unique to each instance! examples ould beinstance_name! thread  and undo_tablespace

!entical parameters-arameters in this category must be the same "or each instance! examplesould be db_name and control_file

Neit,er uni:ue ori!entical parameters

parameters that are not in any o" the above! examples ould be db_cache_size,large_pool_size, local_listener  and gcs_servers_processes

*he main unique parameters that you should +no about are

• instance0name  de"ines the name o" the .racle instance (de"ault is the value o" the oracle_sid

variable)• instance0numer   a unique number "or each instance must be greater than but smaller than

the max_instance parameter

• t,rea!  speci"ies the set o" redolog "iles to be used by the instance

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 11/47

• un!o0talespace  speci"ies the name o" the undo tablespace to be used by the instance

• rollack0se+ments  you should use  Automatic 3ndo :anagement

• cluster0interconnects  use i" only i" .racle has trouble not pic+ing the correct interconnects

*he identical unique parameters that you should +no about are belo you can use the belo query tovie all o" them

  select name! isinstanceDmodi"iable "rom vJparameter here isinstanceDmodi"iable K 0"alse0 order byname@

• cluster0!ataase  options are true or false! mounts the control "ile in either share (cluster) orexclusive mode! use "alse in the belo cases

o onverting "rom no archive log mode to archive log mode and vice versa

o nabling the "lashbac+ database "eature

o -er"orming a media recovery on a system table

o :aintenance o" a node

• actie0instance0count  used "or primary1secondary <A environments

• cluster0!ataase0instances  speci"ies the number o" instances that ill be accessing thedatabase (set to maximum L o" nodes)

• !ml0locks  speci"ies the number o" 6:? loc+s "or a particular instance (only change i" you get.<A'' errors)

• +c0files0to0locks  speci"y the number o" global loc+s to a data "ile! changing this disables theache /usion#

• ma;0commit0propa+ation0!elay  in"luences the mechanism .racle uses to synchroni9e theS= among all instances

• instance0+roups  speci"y multiple parallel query execution groups and assigns the currentinstance to those groups

• parallel0instance0+roup  speci"ies the group o" instances to be used "or parallel queryexecution

• +cs0serer0processes  speci"y the number o" loc+ manager server (?:S) bac+groundprocesses used by the instance "or ache /usion

• remote0listener   register the instance ith listeners on remote nodes#

syntax "orparameter "ile

MinstanceDnameN#MparameterDnameNKMparameterDvalueN

inst%#dbDcacheDsi9e K %O#undoDmanagementKauto

examplealter system set dbD2+DcacheDsi9eK%m scopeKsp"ile sidK0inst%0@

=oteE use the sid option to speci"y a particular instance

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 12/47

$tartin+ an! $toppin+ nstances

*he srvctl  command is used to start1stop an instance! you can also use slplus to start and stop theinstance

start all instances

srvctl start database d MdatabaseN o MoptionN

=oteE starts listeners i" not already running! you can use the o option to speci"ystartup1shutdon options! see belo "or options

"orceopenmountnomount

stop all instances

srvctl stop database d MdatabaseN o MoptionN

=oteE the listeners are not stopped! you can use the o option to speci"ystartup1shutdon options! see belo "or options

immediateabortnormaltransactional

start1stop particularinstance

srvctl PstartQstopR database d MdatabaseN i MinstanceN!MinstanceN

'n!o 1ana+ement

;nstances in a <A do not share undo! they each have a dedicated undo tablespace# 3sing theundo_tablespace parameter each instance can point to its on undo tablespace

un!o talespaceinstance%#undoDtablespaceKundoDtbs%instance2#undoDtablespaceKundoDtbs2

emporary alespace

;n a <A environment you should setup a temporary tablespace group@ this group is then used by allinstances o" the <A# ach instance creates a temporary segment in the temporary tablespace it isusing# ;" an instance is running a large sort! temporary segments can be reclaimed "rom segments "romother instances in that tablespace#

useful iews

+<sort0se+ment  explore current and maximum sort segment usage statistics (chec+

columns "reedDextents! "reeDrequests !i" they gro increase tablespace si9e)+<tempse+0usa+e  explore tempry segment usage details such as name! S5?! etc<tempfile - i!entify  temporary data"iles being used "or the temporary tablespace

Re!olo+s

; have already discussed redologs! in a <A environment every instance has its on set o" redologs#ach instance has exclusive rite access to its on redologs! but each instance can read each othersredologs! this is used "or recovery# <edologs are located on the shared storage so that all instances can

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 13/47

have access to each others redologs# *he process is a little di""erent to the standard .racle henchanging the archive mode

archive mode(<A)

S5?N alter system set cluster0!ataase=false scope=spfile si!=>pro!3>?srvctl stop database d MdatabaseNS5?N startup mount

S5?N alter database archivelog@S5?N alter system set cluster0!ataase=true scope=spfile si!=>pro!3>?S5?N shutdon@srvctl start database d prod

#las,ack

 Again ; have already tal+ed about "lashbac+! there is no di""erence in <A environment apart "rom thesetting up

"lashbac+ (<A)

LL :a+e sure that the database is running in archive log modeS5?N archive log list

LL Setup the "lashbac+S5?N alter system set cluster0!ataase=false scope=spfile si!=>pro!3>?S5?N alter system set 6>D<.8<,D/;?D6S*DS;K2: scopeKsp"ile@S5?N alter system set 6>D<.8<,D/;?D6S*K01oc"s21"lashbac+0 scopeKsp"ile@srvctl stop database p prod%S5?N startup mountS5?N alter database "lashbac+ on@S5?N shutdon@srvctl start database p prod%

$R6C comman!

We have already come across the srvctl above@ this command is called the server control utility# ;t candivided into to categories

• 6atabase con"iguration tas+s

• 6atabase instance control tas+s

.racle stores database con"iguration in a repository! the con"iguration is stored in the .racle luster<egistry (.<) that as created hen <A as installed! it ill be located on the shared storage# Srvctluses <S to communicate and per"orm startup and shutdon commands on other nodes#

; suggest that you loo+ up the command but ; ill provide a "e examples

display theregistereddatabases

srvctl con"ig database

status srvctl status database d Mdatabasesrvctl status instance d MdatabaseN i MinstanceNsrvctl status nodeapps n MnodeN

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 14/47

srvctl status service d MdatabaseNsrvctl status asm n MnodeN

stopping1starting

srvctl stop database d MdatabaseNsrvctl stop instance d MdatabaseN i MinstanceN!MinstanceNsrvctl stop service d MdatabaseN Ps MserviceNMserviceNR Pi MinstanceN!MinstanceNRsrvctl stop nodeapps n MnodeN

srvctl stop asm n MnodeN

srvctl start database d MdatabaseNsrvctl start instance d MdatabaseN i MinstanceN!MinstanceNsrvctl start service d MdatabaseN s MserviceNMserviceN i MinstanceN!MinstanceNsrvctl start nodeapps n MnodeNsrvctl start asm n MnodeN

adding1removing

srvctl add database d MdatabaseN o MoracleDhomeNsrvctl add instance d MdatabaseN i MinstanceN n MnodeNsrvctl add service d MdatabaseN s MserviceN r Mpre"erredDlistNsrvctl add nodeapps n MnodeN o MoracleDhomeN A MnameQipN1netor+srvctl add asm n MnodeN i MasmDinstanceN o MoracleDhomeN

srvctl remove database d MdatabaseN o MoracleDhomeNsrvctl remove instance d MdatabaseN i MinstanceN n MnodeNsrvctl remove service d MdatabaseN s MserviceN r Mpre"erredDlistNsrvctl remove nodeapps n MnodeN o MoracleDhomeN A MnameQipN1netor+srvctl asm remove n MnodeN

$erices

Services are used to manage the or+load in .racle <A! the important "eatures o" services are

• used to distribute the or+load

• can be con"igured to provide high availability

• provide a transparent ay to direct or+load

*he vie v!services contains in"ormation about services that have been started on that instance! here isa list "rom a "resh <A installation

*he table above is described belo

• /oal  allos you to de"ine a service goal using service time! throughput or none

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 15/47

• Connect ime oa! *alancin+ /oal  listeners and midtier servers contain current in"ormationabout service per"ormance

• Distriute! ransaction 5rocessin+  used "or distributed transactions

• A0"A0Notifications  in"ormation about nodes being up or don ill be sent to midtier servers

via the advance queuing mechanism

• 5referre! an! Aailale nstances  the pre"erred instances "or a service! available ones are thebac+up instances

,ou can administer services using the "olloing tools

• 6>A

• : (nterprise :anager)

• 6>:SDS<8;S

• Server ontrol (srvctl)

*o services are created hen the database is "irst installed@ these services are running all the time andcannot be disabled#

• sys<ack+roun!  used by an instance0s bac+ground processes only

• sys<users  hen users connect to the database ithout speci"ying a service they use thisservice

add

srvctl add service d 6% s >A*HDS<8; r node%!node2 a node$

=oteE the options are describe belo

d databases the servicer the service ill running on the these nodesa i" nodes in the r list are not running then run on this node

remove srvctl remove service d 6% s >A*HDS<8;

start srvctl start service d 6% s >A*HDS<8;

stop srvctl stop service d 6% s >A*HDS<8;

status srvctl status service d 6% s >A*HDS<8;

service (example) LL create the G.> class>C;= 6>:SDSH63?<#createDobDclass(

  obDclassDname KN 0>A*HDG.>D?ASS0!  service KN 0>A*HDS<8;0)@=6@1

LL Crant the privileges to execute the obgrant execute on sys#batchDobDclass to vallep@

LL create a ob associated ith a ob class

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 16/47

>C;= 6>:SDSH63?<#createDob(  obDname KN 0myDuser#batchDobDtest0!  obDtype KN 0-?S5?D>?.F0!  obDaction KN S,S*;:S*A:-0  repeatDinterval KN 0/<5K6A;?,@0!

  obDclass KN 0S,S#>A*HDG.>D?ASS0!  endDdate KN =3??!  enabled KN *<3!  comments KN 0*est batch ob to sho <A services0)@=6@1

LL assign a ob class to an existing obexec dbmsDscheduler#setDattribute(0:,D>A*HDG.>0! 0G.>D?ASS0!0>A*HDG.>D?ASS0)@

Cluster Rea!y $erices (CR$)

<S is .racle0s clusterare so"tare@ you can use it ith other thirdparty clusterare so"tare! though itis not required (apart "rom H- *rue&4)#

<S is start automatically hen the server starts@ you should only stop this service in the "olloingsituations

•  Applying a patch set to J.<AD<SDH.:

• .1S maintenance

• 6ebugging <S problems

CR$ A!ministration

starting

LL Starting <S using .racle %g <%not possible

LL Starting <S using .racle %g <2J.<AD<SDH.:1bin1crsctl start crs

stopping

LL Stopping <S using .racle %g <%srvctl stop d database MdatabaseNsrvctl stop asm n MnodeNsrvctl stop nodeapps n MnodeN1etc1init#d1init#crs stop

LL Stopping <S using .racle %g <2

J.<AD<SDH.:1bin1crsctl stop crs

disabling1enabling

LL stop <S restarting a"ter a reboot! basically permanent over reboots

LL .racle %g <%1etc1init#d1init#crs PdisableQenableR

LL .racle %g <2J.<AD<SDH.:1bin1crsctl PdisableQenableR crs

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 17/47

chec+ing

J.<AD<SDH.:1bin1crsctl chec+ crsJ.<AD<SDH.:1bin1crsctl chec+ evmdJ.<AD<SDH.:1bin1crsctl chec+ cssdJ.<AD<SDH.:1bin1crsctl chec+ crsdJ.<AD<SDH.:1bin1crsctl chec+ install ait &

Resource Applications (CR$ 'tilities)

status

J.<AD<SDH.:1bin1crsDstatJ.<AD<SDH.:1bin1crsDstat tJ.<AD<SDH.:1bin1crsDstat lsJ.<AD<SDH.:1bin1crsDstat p

=oteEt more readable displayls permission listingp parameters

create pro"ile J.<AD<SDH.:1bin1crsDpro"ile

register1unregister applicationJ.<AD<SDH.:1bin1crsDregister J.<AD<SDH.:1bin1crsDunregister 

Start1Stop an applicationJ.<AD<SDH.:1bin1crsDstartJ.<AD<SDH.:1bin1crsDstop

<esource permissionsJ.<AD<SDH.:1bin1crsDgetparamJ.<AD<SDH.:1bin1crsDsetparam

<elocate a resource J.<AD<SDH.:1bin1crsDrelocate

No!es

member number1nameolsnodes n

=oteE the olsnodes command is located in J.<AD<SDH.:1bin

local node name olsnodes l

activates logging olsnodes gOracle nterfaces

display oi"c"g geti"

delete oic"g delig global

setoic"g seti" global Minter"ace nameN1MsubnetNEpublicoic"g seti" global Minter"ace nameN1MsubnetNEclusterDinterconnect

/loal $erices Daemon Control

starting gsdctl start

stopping gsdctl stop

status gsdctl status

Cluster Confi+uration (clscf+ is use! !urin+ installation)

create a ne con"igurationclsc"g install

=oteE the clsc"g command is located in J.<AD<SDH.:1bin

upgrade or dongrade andexisting con"iguration

clsc"g upgradeclsc"g Tdongrade

add or delete a node "rom thecon"iguration

clsc"g addclsc"g Tdelete

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 18/47

create a special singlenodecon"iguration "or AS:

clsc"g Tlocal

brie" listing o" terminology usedin the other nodes

clsc"g Tconcepts

used "or tracing clsc"g Ttrace

help clsc"g hCluster Name C,eck

print cluster name

cemutlo n

=oteE in .racle Bi the ulity as called UcemutlsU! the command is locatedin J.<AD<SDH.:1bin

print the clusterare versioncemutlo

=oteE in .racle Bi the ulity as called UcemutlsU

No!e $cripts

 Add =ode

addnode#sh

=oteE see adding and deleting nodes 

6elete =odedeletenode#sh

=oteE see adding and deleting nodes 

Oracle Cluster Re+istry (OCR)

 As you already +no the .< is the registry that contains in"ormation

• =ode list

• =ode membership mapping

• 6atabase instance! node and other mapping in"ormation

• haracteristics o" any thirdparty applications controlled by <S

*he "ile location is speci"ied during the installation! the "ile pointer indicating the .< device location isthe ocr#loc! and this can be in either o" the "olloing

• linux 1etc1oracle

• solaris 1var1opt1oracle

*he "ile contents loo+ something li+e belo! this as ta+en "rom my installation

orc#lococrcon"igDlocK1u21oradata1racdb1.</ileocrmirrorcon"igDlocK1u21oradata1racdb1.</ileDmirror localDonlyK/A?S

.< is import to the <A environment and any problems must be immediately actioned! the commandcan be "ound in located in J.<AD<SDH.:1bin

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 19/47

OCR 'tilities

log "ile J.<ADH.:1log1MhostnameN1client1ocrcon"igDMpidN#log

chec+ing

ocrchec+

=oteE ill return the .< version! total space allocated! space used! "ree space!

location o" each device and the result o" the integrity chec+

dump contents

ocrdump

=oteE by de"ault it dumps the contents into a "ile named .<63:-/;? in thecurrent directory

export1importocrcon"ig export M"ileN

ocrcon"ig restore M"ileN

bac+up1restore

L sho bac+upsocrcon"ig shobac+up

L to change the location o" the bac+up! you can even speci"y a AS: dis+ocrcon"ig bac+uploc MpathQVasmN

L per"orm a bac+up! ill use the location speci"ied by the bac+uploc locationocrcon"ig manualbac+up

L per"orm a restoreocrcon"ig restore M"ileN

L delete a bac+uporccon"ig delete M"ileN

=oteE there are many more option so see the ocrcon"ig man page

add1remove1replace

LL add1relocate the ocrmirror "ile to the speci"ied location

ocrcon"ig replace ocrmirror 01oc"s21ocr2#db"0

LL relocate an existing .< "ileocrcon"ig replace ocr 01oc"s%1ocrDne#db"0

LL remove the .< or .<:irror "ileocrcon"ig replace ocr ocrcon"ig replace ocrmirror

6otin+ Disk

*he voting dis+ as ; mentioned in the architecture is used to resolve membership issues in the event o" apartitioned cluster! the voting dis+ protects data integrity#

querying crsctl query css votedis+

adding crsctl add css votedis+ M"ileN

deleting crsctl delete css votedis+ M"ileN

<A >ac+ups and <ecovery

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 20/47

>ac+ups and recovery is very similar to a single instance database# *his article covers only the speci"icissues that surround <A bac+ups and recovery#

>ac+ups can be di""erent depending on the si9e o" the company

• small company may use tools such as tar! cpio! rsync

• medium1large company 8eritas =etbac+up! <:A=

• nterprise company SA= mirroring ith a bac+up option li+e =etbac+up or <:A=

.racle <A can use all the above bac+up technologies! but .racle pre"ers you to use <:A= oracle onbac+up solution#

*ackup *asics

.racle bac+ups can be ta+en hot or cold! a bac+up ill comprise o" the "olloing

• 6ata"iles

• ontrol /iles

•  Archive redolog "iles

• -arameter "iles (init#ora or S-/;?)

6atabases have no gron to very large si9es ell over a terabyte in si9e in some cases! thus tapesbac+ups are not used in these cases but sophisticated dis+ mirroring have ta+en their place# <:A= canbe used in either a tape or dis+ solution! it can even or+ ith thirdparty solutions such as 8eritas=etbac+up#

;n a .racle <A environment it is critical to ma+e sure that all archive redolog "iles are located on sharedstorage! this is required hen trying to recover the database! as you need access to all archive redologs#

<:A= can use parallelism hen recovering! the node that per"orms the recovery must have access to allarchived redologs! hoever! during recovery only one node applies the archived logs as in a standardsingle instance con"iguration#

.racle <A also supports .racle 6ata Cuard! thus you can have a primary database con"igured as a<A and a standby database also con"igured as a <A#

nstance Recoery

;n a <A environment there are to types o" recovery

• Cras, Recoery  means that all instances have "ailed! thus they all need to be recovered

• nstance Recoery  means that one or more instances have "ailed# this instance can then berecovered by the surviving instances

<edo in"ormation generated by an instance is called a thread of redo# All log "iles "or that instance belongto this thread! an online redolog "ile belongs to a group and the group belongs to a thread# 6etails aboutlog group "ile and thread association details are stored in the control "ile# <A databases have multiplethreads o" redo! each instance has one active thread! and the threads are parallel timelines and together"orm a stream# A stream consists o" all the threads o" redo in"ormation ever recorded! the streams "ormthe timeline o" changes per"ormed to the database#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 21/47

.racle records the changes made to a database! these are called change vectors# ach vector is adescription o" a single change! usually a single bloc+# A redo record contains one or more change vectorsand is located by its "edo #$te %ddress (<>A) and points to a speci"ic location in the redolog "ile (orthread)# ;t ill consist o" three components

• log sequence number 

• bloc+ number ithin the log

• byte number ithin the bloc+

hec+points are the same in a <A environment and a single instance environment! ; have alreadydiscussed chec+points! hen a chec+point needs to be triggered! .racle ill loo+ "or the threadchec+point that has the loest chec+point S=! all bloc+s in memory that contain changes made prior tothis S= across all instances must be ritten out to dis+# ; have discussed ho to control recovery in my.racle section and this applies to <A as ell#

Cras, Recoery

rash recovery is basically the same "or a single instance and a <A environment! ; have a completerecovery section in my .racle section! here is a note detailing the di""erence#

/or a single instance the "olloing is the recovery process

%# *he ondis+ bloc+ is the starting point "or the recovery@ .racle ill only consider the bloc+ on thedis+ so the recovery is simple# rash recovery ill automatically happen using the online redologs that are current or active

2# *he starting point is the last "ull chec+point# *he starting point is provided by the control "ile andcompared against the same in"ormation in the data "ile headers! only the changes need to beapplied

$# *he bloc+ speci"ied in the redolog is read into cache! i" the bloc+ has the same timestamp as theredo record (S= match) the redo is applied#

/or a <A instance the "olloing is the recovery process

%# A "oreground process in a surviving instance detects an Uinvalid bloc+ loc+U condition hen anattempt is made to read a bloc+ into the bu""er cache# *his is an indication that an instance has"ailed (died)

2# *he "oreground process sends a noti"ication to instance system monitor (S:.=) hich begin tosearch "or dead instances# S:.= maintains a list o" all the dead instances and invalid bloc+loc+s# .nce the recovery and cleanup has "inished this list is updated#

$# *he death o" another instance is detected i" the current instance is able to acquire that instance0sredo thread loc+s! hich is usually held by an open and active instance#

.racle <A uses a topass recovery! because a data bloc+ could have been modi"ied in any o" theinstances (dead or alive)! so it needs to obtain the latest version o" the dirty bloc+ and it uses -; (-ast;mage) and >loc+ Written <ecord (>W<) to archive this in a quic+ and timely "ashion#

*lock@rittenRecor!(*R@)

*he cache aging and incremental chec+point system ould rite a number o" bloc+s to dis+!hen the 6>W< completes a data bloc+ rite operation! it also adds a redo record thatstates the bloc+ has been ritten (data bloc+ address and S=)# 6>Wn can rite bloc+ritten records (>W<s) in batches! though in a la9y "ashion# ;n <A a >W< is ritten hen

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 22/47

an instance rites a bloc+ covered by a global resource or hen it is told that its past image(-;) bu""er it is holding is no longer necessary#

5ast ma+e

(5)

*his is as ma+es <A cache "usion or+@ it eliminates the rite1rite contention problemthat existed in the .-S database# A -; is a copy o" a globally dirty bloc+ and is maintained inthe database bu""er cache@ it can be created and saved hen a dirty bloc+ is shipped acrossto another instance a"ter setting the resource role to global# *he CS is responsible "or

in"orming an instance that its -; is no longer needed a"ter another instance rites a neer(current) version o" the same bloc+# -;0s are discarded hen CS posts all the holdinginstances that a ne and consistent version o" that particular bloc+ is no on dis+#

*he first pass does not per"orm the actual recovery but merges and reads redo threads to create a hashtable o" the bloc+s that need recovery and that are not +non to have been ritten bac+ to the data"iles#*he chec+point S= is need as a starting point "or the recovery@ all modi"ied bloc+s are added to therecovery set (an organi9ed hash table)# A bloc+ ill not be recovered i" its >W< version is greater than thelatest -; in any o" the bu""er caches#

*he secon! pass S:.= rereads the merged redo stream (by S=) "rom all threads needing recovery!

the redolog entries are then compared against a recovery set built in the "irst pass and any matches areapplied to the inmemory bu""ers as in a single pass recovery# *he bu""er cache is "lushed and thechec+point S= "or each thread is updated upon success"ul completion#

Cac,e #usion Recoery

; have a detailed section on cache "usion! this section covers the recovery! cache "usion is only used in<A environments! as additional steps are required! such as C<6 recon"iguration! internodescommunication! etc# *here are to types o" recovery

• rash <ecovery all instances have "ailed

• ;nstance <ecovery one instance has "ailed

;n both cases the threads "rom "ailed instances need to be merged@ in an instance recovery S:.= illper"orm the recovery here as in a crash recovery a "oreground process per"orms the recovery#

*he main "eatures (advantages) o" cache "usion recovery are

• <ecovery cost is proportional to the number o" "ailures! not the total number o" nodes

• ;t eliminates dis+ reads o" bloc+s that are present in a surviving instance0s cache

• ;t prunes recovery set based on the global resource loc+ state

• *he cluster is available a"ter an initial log scan! even be"ore recovery reads are complete

;n cache "usion the starting point "or recovery o" a bloc+ is its most current -; version! this could belocated on any o" the surviving instances and multiple -; bloc+s o" a particular bu""er can exist#

Remasterin+ is the term used that describes the operation hereby a node attempting recovery tries toon or master the resource(s) that ere once mastered by another instance prior to the "ailure# When oneinstance leaves the cluster! the C<6 o" that instance needs to be redistributed to the surviving nodes#<A uses an algorithm called laz$ remastering  to remaster only a minimal number o" resources during arecon"iguration# *he entire -arallel ache :anagement (-:) loc+ space remains invalid hile the 6?:and S:.= complete the belo steps

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 23/47

%# ;6?: master node discards loc+s that are held by dead instances! the space is reclaimed by thisoperation is used to remaster loc+s that are held by the surviving instance "or hich a deadinstance as remastered

2# S:.= issues a message saying that it has acquired the necessary bu""er loc+s to per"ormrecovery

?ets loo+ at an example on hat happens during a remastering! lets presume the "olloing

• ;nstance A masters resources %! $! ' and 7

• ;nstance > masters resources 2! 4! &! and

• ;nstance masters resources B! %! %% and %2

;nstance > is removed "rom the cluster! only the resources "rom instance > are evenly remastered acrossthe surviving nodes (no resources on instances A and are a""ected)! this reduces the amount o" or+the <A has to per"orm! li+eise hen a instance oins a cluster only minimum amount o" resources areremastered to the ne instance#

*efore Remasterin+ After Remasterin+

,ou can control the remastering process ith a number o" parameters

 0+cs0fast0confi+ enables "ast recon"iguration "or gcs loc+s (trueQ"alse)

 0lm0master0wei+,t controls hich instance ill hold or (re)master more resources than others

 0+cs0resources controls the number o" resources an instance ill master at a time

you can also "orce a dynamic remastering (6<:) o" an obect using oradebug

"orce dynamicremastering (6<:)

LL .btain the .>G*D;6 "orm the belo tableS5?N select O "rom vJgcsp"masterDin"o@

LL 6etermine ho masters it

S5?N oradebug setmypidS5?N oradebug l+debug a M.>G*D;6N

LL =o remaster the resourceS5?N oradebug setmypidS5?N oradebug l+debug m p+ey M.>G*D;6N

*he steps o" a C<6 recon"iguration is as "ollos

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 24/47

• ;nstance death is detected by the cluster manager 

• <equest "or -: loc+s are "ro9en

• nqueues are recon"igured and made available

• 6?: recovery

• CS (-: loc+) is remastered

• -ending rites and noti"ications are processed

• ; -ass recovery

o *he instance recovery (;<) loc+ is acquired by S:.=

o *he recovery set is prepared and built! memory space is allocated in the S:.= -CA

o S:.= acquires loc+s on bu""ers that need recovery

• ;; -ass recovery

o ;; pass recovery is initiated! database is partially available

o >loc+s are made available as they are recovered

o *he ;< loc+ is released by S:.=! recovery is then complete

o *he system is available

Craphically it loo+s li+e belo

RAC 5erformance

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 25/47

;n this section ; ill mainly discuss .racle <A tuning# /irst let0s revie the best practices o" .racledesign regarding the application and database

• .ptimi9e connection management! ensure that the middle tier and programs that connect to thedatabase are e""icient in connection management and do not log on or o"" repeatedly

• *une the S5? using the available tools such as A66: and S5? *uning Advisor 

• nsure that applications use in! ariales! cursor_sharing  as introduced to solve this problem

• 3se pac+ages and procedures (because they are compiled) in place o" anonymous -?1S5?bloc+s and big S5? statements

• 3se locally managed tablespaces and automatic segment space management to helpper"ormance and simpli"y database administration

• 3se automatic undo management and temporary tablespace to simpli"y administration andincrease per"ormance

• nsure you use large caching hen using sequences! unless you cannot a""ord to lose sequence

during a crash

•  Avoid using 66? in production! it increases invalidations o" the already parsed S5? statementsand they need to be recompiled

• -ortion tables and indexes to reduce index lea" contention (bu""er busy global cr problems)

• .ptimi9e contention on data bloc+s (hot spots) by avoiding small tables ith too many ros in abloc+

=o e can revie <A speci"ic best practices

• onsider using application partitioning (see belo)

• onsider restricting 6:?intensive users to using one instance! thus reducing cache contention

• Feep readonly tablespaces aay "rom 6:?intensive tablespaces! they only require minimumresources thus optimi9ing ache /usion per"ormance

•  Avoid auditing in <A! this causes more shared library cache loc+s

• 3se "ull tables scans sparingly! it causes the CS to service lots o" bloc+ requests! see tablev!s$sstat column Utable scans (long tables)U

• i" the application uses lots o" logins! increase the value o" s$s.audsess!  sequence

5artitionin+ @orkloa!

Wor+load partitioning is a certain type o" or+load that is executed on an instance! that is partitioningallos users ho access the same set o" data to log on to the same instance# *his limits the amount o"data that is shared beteen instances thus saving resources used "or messaging and ache /usion databloc+ trans"er#

,ou should consider the "olloing hen deciding to implement partitioning

• ;" the -3 and private interconnects are o" high per"ormance then there is no need to partition

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 26/47

• -artitioning does add complexity! thus i" you can increase -3 and the interconnect per"ormancethe better 

• .nly partition i" per"ormance is betting impacted

• *est both partitioning and nonpartitioning to hat di""erence it ma+es! then decide i" partitioning is

orth it

RAC @ait ents

 An event is an operation or particular "unction that the .racle +ernel per"orms on behal" o" a user or a.racle bac+ground process! events have speci"ic names li+e database event # Whenever a session has toait "or something! the ait time is trac+ed and charged to the event that as associated ith that ait#vents that are associated ith all such aits are +non as &ait events# *hey are a number o" aitclasses

• ommit

• Scheduler 

•  Application

• on"iguration

• 3ser ;1.

• System ;1.

• oncurrency

• =etor+

•  Administrative

• luster 

• ;dle

• .ther 

*here are over di""erent events spread across the above list! hoever you probably ill only deal ithabout ' or so that can improve per"ormance#

When a session requests access to a data bloc+ it sends a request to the loc+ master "or properauthori9ation! the request does not +no i" it ill receive the bloc+ via ache /usion or a permission toread "rom the dis+# *o placeholder events

• global cache cr reuest 'consistent read - cr(

• global cache curr reuest 'current - curr(

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 27/47

+eep trac+ o" the time a session spends in this state# *here are number o" types o" ait events regardingaccess to a data bloc+

@ait entContention

typeDescription

+c currentlock -way

rite1rite

 An instance requests authori9ation "or a bloc+ to be accessed in current mode tomodi"y a bloc+! the instance mastering the resource receives the request# *hemaster has the current version o" the bloc+ and sends the current copy o" thebloc+ to the requestor via ache /usion and +eeps a -ast ;mage (#-;)

;" you get this then do the "olloing

•  Analy9e the contention! segments in the Ucurrent bloc+s receivedU sectiono" AW<

• 3se application partitioning scheme

• :a+e sure the system has enough -3 poer 

•

:a+e sure the interconnect is as "ast as possible

• nsure that soc+et send and receive bu""ers are con"igured correctly

 

+c currentlock B-way

rite1rite

an instance requests authori9ation "or a bloc+ to be accessed in current mode tomodi"y a bloc+! the instance mastering the resource receives the request and"orards it to the current holder o" the bloc+! as+ing it to relinquish onership#*he holding instance sends a copy o" the current version o" the bloc+ to therequestor via ache /usion and trans"ers the exclusive loc+ to the requestinginstance# ;t also +eeps a past ;mage (-;)#

3se the above actions to increase the per"ormance

+c currentlock -way

rite1read*he di""erence ith the one above is that this sends a copy o" the bloc+ thus+eeping the current copy#

+c currentlock B-way

rite1read*he di""erence ith the one above is that this sends a copy o" the bloc+ thus+eeping the current copy#

+c current

lock usy

rite1rite

*he requestor ill eventually get the bloc+ via cache "usion but it is delayed dueto one o" the "olloing

• *he bloc+ as being used by another session on another session

• as delayed as the holding instance could not rite the correspondingredo record immediately

;" you get this then do the "olloing

• nsure the log riter  is tuned

+c currentufferusy

local*his is the same as above (gc current bloc+ busy)! the di""erence is that anothersession on the same instance also has requested the bloc+ (hence localcontention)

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 28/47

+c currentlockcon+este!

none*his is caused i" heavy congestion on the CS! thus -3 resources arestretched

n:ueue unin+

.racle <A uses a queuing mechanism to ensure proper use o" shared resources! it is called Clobalnqueue Services (CS)# nqueue ait is the time spent by a session aiting "or a shared resource!here are some examples o" enqueuesE

• updating the control "ile (/ enqueue)

• updating an individual ro (*I enqueue)

• exclusive loc+ on a table (*: enqueue)

nqueues can be managed by the instance itsel" others are used globally! CS is responsible "orcoordinating the global resources# *he "ormula used to calculate the number o" enqueue resources is asbelo

  CS <esources K 6>D/;?S V 6:?D?.FS V =533D<S.3<S V -<.SS V*<A=SA*;.= x (% V (= %)1=)

  = K number o" <A instances

displayingenqueues stats

S5?N column currentDutili9ation heading currentS5?N column maxDutili9ation heading maxDusageS5?N column initialDallocation heading initialS5?N column resourceDlimit "ormat a2$@

S5?N select O "rom vJresourceDlimit@

A@R an! RAC

/rom a <A point o" vie there are a number o" <Aspeci"ic sections that you need to loo+ at in the AW<! in the report section is a AW< o" my home <A environment! you can vie the belo report#

RAC A@R$ection

Report Description

Numer ofnstances

instances lists the number o" instances "rom the beginning and end o" the AW< report

nstance +loalcac,e loa!

profile

globalcache

in"ormation about the interinstance cache "usion data bloc+ and messagingtra""ic! because my  AW< report is lighteight here is a more heavy used

<A example

Clobal ache ?oad -ro"ile -er Second -er *ransaction  Clobal ache bloc+s receivedE $%'#$7 %2#2Clobal ache bloc+s servedE 24#$ B#&7CS1CS messages receivedE '2'#%& 2#%CS1CS messages sentE 7&'#$2 $#B%

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 29/47

*he "irst to statistics indicate the number o" bloc+s trans"erred to or "romthis instance! thus i" you are using a F bloc+ si9e

  SentE 24 x !%B2 K %B&& bytes1sec K 2# :>1sec  <eceivedE $%' x !%B2 K 2'4 bytes1sec K 2#& :>1sec

to determine the amount o" netor+ tra""ic generated due to messaging you"irst need to "ind the average message si9e (this as %B$ on my system)

select sum(+xmsi9e O (+xmrcv V +xmsnt V +xmqsnt)) 1 sum((+xmrcv V+xmsnt V +xmqsnt)) Uavg :essage si9eU "rom xJ+xm

here +xmrcv N or +xmsnt N or +xmqsnt N @

then calculate the amount o" messaging tra""ic on this netor+

  %B$ (7&' V '2') K $7 K #4 :>

to calculate the total netor+ tra""ic generated by cache "usion

  K 2# V 2#& V #4 K ' :>ytes1sec  K ' x K 4 :bits1sec

*he 6>W< /usion rites statistic indicates the number o" times the local6>W< as "orced to rite a bloc+ to dis+ due to remote instances! thisnumber should be low.

/local cac,eefficiency

percenta+e

globalcache

e""iciency

this section shos ho the instance is getting all the data bloc+s it needs#*he best order is the "olloingE

• ?ocal cache

• <emote cache

•

6is+

*he "irst to give the cache hit ratio "or the instance! you are loo+ing "or avalue less than %X! i" you are getting higher values then you may consider application partitioning# 

/C$ an! /$ -workloa!c,aracteristics

CS andCS

or+load

this section contains timing statistics "or global enqueue and global cache# As a general rule you are loo+ing "or

•  All timings related to < (onsistent <ead) processing bloc+should be less than % msec

•  All timings related to 3<<=* bloc+ processing should be less

than 2 msec

1essa+in+statistics

messaging

*he "irst section relates to sending a message and should be less than %second#

*he second section details the brea+up o" direct and indirect messages!direct messages are sent by a instance "oreground or the user processes toremote instances! indirect are messages that are not urgent and are pooledand sent#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 30/47

$ericestatistics

Servicestats

shos the resources used by all the service instance supports

$erice waitclass statistics

Service aitclass 

summari9es aits in di""erent categories "or each service

op 4 CR an!

current lockse+ements

*op ' <

and currentbloc+s

conatns the names o" the top ' contentious segments (table or index)# ;" a

table or index has a very high percentage o" < and urrent bloc+trans"ers you need to investigate# *his is pretty much li+e a normal singleinstance#

Cluster nterconnect

 As ; stated above the interconnect it a critical part o" the <A! you must ma+e sure that this is on the besthardare you can buy# ,ou can con"irm that the interconnect is being used in .racle Bi and %g by usingthe command oradebug  to dump in"ormation out to a trace "ile! in .racle %g <2 the cluster interconnectis also contained in the alert.log  "ile! you can vie my in"ormation "rom here#

interconnect

S5?N oradebug setmypid

S5?N oradebug ipc

=oteE loo+ in the userDdumpDdest directory! the trace ill be there

Clobal <esource 6irectory (C<6)

*he <A environment includes many resources such as multiple versions o" data bloc+ bu""ers in bu""ercaches in di""erent modes! .racle uses loc+ing and queuing mechanisms to coordinate loc+ resources!data and interinstance data requests# <esources such as data bloc+s and loc+s must be synchroni9edbeteen nodes as nodes ithin a cluster acquire and release onership o" them# *he synchroni9ationprovided by the Clobal <esource 6irectory (C<6) maintains a cluster ide concurrency o" the resourcesand in turn ensures the integrity o" the shared data# Synchroni9ation is also required "or bu""er cachemanagement as it is divided into multiple caches! and each instance is responsible "or managing its onlocal version o" the bu""er cache# opies o" data are exchanged beteen nodes! this sometimes isre"erred to as the global cache but in reality each nodes bu""er cache is separate and copies o" bloc+s areexchanged through traditional distributed loc+ing mechanism#

Clobal ache Services (CS) maintain the cache coherency across bu""er cache resources and Clobalnqueue Services (CS) controls the resource management across the clusters nonbu""er cacheresources#

Cac,e Co,erency

ache coherency identi"ies the most uptodate copy o" a resource! also called the master  copy! it uses amechanism by hich multiple copies o" an obect are +eep consistent beteen .racle instances# -arallelache :anagement (-:) ensures that the master copy o" a data bloc+ is stored in one bu""er cacheand consistent copies o" the data bloc+ are stored in other bu""er caches! the process ?Fx is responsible"or this tas+#

*he loc+ and resource structures "or instance loc+s reside in the C<6 (also called the 6?:)! its adedicated area ithin the shared pool# 6etails about the data bloc+s resources and cached versions are

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 31/47

maintained by CS# Additional details such as the location o" the most current version! state o" the bu""er!role o" the data bloc+ (local or global) and onership are maintained by CS# Clobal cache together ithCS "orm the C<6# ach instance maintains a part o" the C<6 in its SCA# *he CS and CS nominateone instance! this ill become the resource master ! to manage all in"ormation about a particularresource# ach instance +nos hich instance master is ith hich resource#

Resources an! n:ueues

 A resource is an identi"iable entity! it has a name or re"erence# *he re"erenced entity is usually a memoryregion! a dis+ "ile! a data bloc+ or an abstract entity# A resource can be oned or loc+ed in various states(exclusive or shared)! all resources are loc+able# A global resource is visible throughout the cluster! thus alocal resource can only be used by the instance at it is local too# ach resource can have a list o" loc+scalled the grant ueue! that are currently granted to users# A convert ueue is a queue o" loc+s that areaiting to be converted to particular mode! this is the process o" changing a loc+ "rom one mode toanother! even a =3?? is a loc+# A resource has a loc+ value bloc+ (?8>)# *he Clobal <esource :anager(C<:) +eeps the loc+ in"ormation valid and correct across the cluster#

?oc+s are placed on a resource grant or a convert queue! i" the loc+ changes it moves beteen thequeues# A loc+ leaves the convert queue under the "olloing conditions

• *he process requests the loc+ termination (it remove the loc+)

• *he process cancels the conversion! the loc+ is moved bac+ to the grant queue in the previousmode

• *he requested mode is compatible ith the most restrictive loc+ in the grant queue and ith allthe previous modes o" the convert queue! and the loc+ is at the head o" the convert queue

onvert requests are processed on a /;/. basis! the grant queue and convert queue are associated itheach and every resource that is managed by the CS#

nqueues are basically loc+s that support queuing mechanisms and that can be acquired in di""erentmodes# An enqueue can be held in exclusive mode by one process and others can hold a nonexclusivemode depending on the type# nqueues are the same in <A as they are in a single instance#

/loal n:ueue $erices (/$)

CS coordinates the requests o" all global enqueues! it also deals ith deadloc+s and timeouts# *hereare to types o" local loc+s! latches and enqueues! latches do not a""ect the cluster only the localinstance! enqueues can a""ect both the cluster and the instance#

nqueues are shared structures that seriali9e access to database resources! they support multiple modesand are held longer than latches! they protect persistent obects such as tables or library cache obects#nqueues can use any o" the "olloing modes

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 32/47

1o!e $ummary Description

N' =3??no access rights! a loc+ is held at this level to indicate that a process is interested ina resource

$$ SubSharedthe resource can be read in an unprotected "ashion other processes can read andrite to the resource! the loc+ is also +non as a ro share loc+

$ Sharedxclusive

the resource can be read and ritten to in an unprotected "ashion! this is also +nonas a <I (ro exclusive) loc+

$ Shareda process cannot rite to the resource but multiple processes can read it# *his is thetraditional share loc+#

$$SubSharedxclusive

.nly one process can hold a loc+ at this level! this ma+es sure that only processescan modi"y it at a time# .ther processes can per"orm unprotected reads# *his is also+no as a S<I (shared ro exclusive) table loc+#

xclusivegrants the holding process exclusive access to the resource! other processescannot read or rite to the resource# *his is also the traditional exclusive loc+#

/loal ocks

ach node has in"ormation "or a set o" resources! .racle uses a hashing algorithm to determine hichnodes hold the directory tree in"ormation "or the resource# Clobal loc+s are mainly o" to types

• ?oc+s used by the CS "or bu""er cache management! these are called -: loc+s

• Clobal loc+s (global enqueue) that .racle synchroni9es ithin a cluster to coordinate non-:resources! they protect the enqueue structures

 An instance ons a global loc+ that protects a resource (i#e# data bloc+ or data dictionary entry) hen theresource enters the instance0s SCA# CSloc+s control access to data "iles (not the data bloc+s) and control "iles and also seriali9e interinstancecommunication# *hey also control library caches and the dictionary cache# xamples o" this are 66?!6:? enqueue table loc+s! transaction enqueues and 66? loc+s or dictionary loc+s# *he S= and mount

loc+ are global loc+s# *ransaction and roloc+s are the same as in a single instance database! the only di""erence is that the enqueues are globalenqueues! ta+e a loo+ in loc+ing "or an in depth vie on ho .racle loc+ing or+s#

1essa+in+

*he di""erence beteen <A and a single instance messaging is that <A uses the high speedinterconnect and a single instance uses shared memory and semaphores! interrupts are used hen oneor more process ant to use the processor in a multiple -3 architecture# CS uses messaging "orinterinstance communication! this is done by messages and asynchronous traps (AS*s)# >oth ?:.= and?:6 use messages to communicate to other instances! the C<6 is updated hen loc+s are required#*he messaging tra""ic can be vieed using the vie 8JCSD:;S#

 A threeay loc+ message involves up to a maximum o" three instances! :aster instance (:)! Holdinginstance (H) and the <equesting instance (<)! the sequence is detailed belo here requesting instance< is interested in bloc+ >% "rom holding instance H# *he resource is mastered in master instance :

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 33/47

%# ;nstance < gets the onership in"ormationabout a resource "rom the C<6! instance< then sends the message to the masterinstance : requesting access to theresource# *his message is sent by a directsend as it is critical

2# ;nstance : receives the message and"orards it to the holding instance H# *hisis also sent directly! this is +non as abloc)ing as$nchronous trap (BAST)#

$# ;nstance H sends the resource to instance<! using the interconnect! the resource iscopied in instance < memory

4# .nce the loc+ handle is obtained on theresource instance < sends anac+noledgment to instance :# *hismessage is queued as it is not critical! thisis called acuisition as$nchronous trap(AAST)#

>ecause CS heavily rely0s on messaging the interconnect must be o" high quality (high per"ormance !lo latency)! also the messages are +ept small (%2 bytes) to increase per"ormance# *he *ra""icontroller (*</) is used to control the 6?: tra""ic beteen the instances in the cluster! it uses bu""eringto accommodate large volumes o" tra""ic# *he *</ +eeps trac+ o" everything by using tic+ets (sequencenumbers)! there is a prede"ined pool o" tic+ets this is dependent on the netor+ send bu""er si9e# A tic+etis obtained be"ore sending any messages! once sent the tic+et is returned to the pool! ?:S or ?:6per"orm this# ;" there are no tic+ets then the message has to ait until a tic+et is available# ,ou can controlthe number o" tic+ets and vie them

systemparameter

 DlmDtic+ets DlmDtic+etDactiveDsendbac+ (used "or aggressive messaging)

tic+et usageselect localDnid local! remoteDnid remote! tc+tDavail avail! tc+tDlimit limit! sndDqDlensendDqueue! tc+tDait aiting "rom vJgesDtra""icDcontroller@

dump tic+etin"ormation

S5?N oradebug setmypidS5?N oradebug unlimitS5?N oradebug l+debug t

=oteE the output can be vieed here 

/loal Cac,e $erices (/C$)

CS loc+s only protect data bloc+s in the global cache (also +no as -: loc+s)! it can be acquired inshare or exclusive mode# ach loc+ element can have the loc+ role set to either local (same as singleinstance) or global# When in global role three loc+ modes are possible! shared! exclusive and null# ;nglobal role mode you can read or rite to the data bloc+ only as directed by the master instance o" thatresource# *he loc+ and state in"ormation is held in the SCA and is maintained by CS! these are calledloc) elements# ;t also holds a chain o" cache bu""er chains that are covered by the corresponding loc+elements# *hese can be vie via vJloc+Delement! the parameter DdbDbloc+DhashDbuc+ets controls thenumber o" hash bu""er chain buc+ets#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 34/47

CS loc+s uses the "olloing modes as stated above

;clusie ()used during update or any 6:? operation! i" another instance requires the bloc+ that has aexclusive loc+ it as+s CS to request that he second instance dison the global loc+

$,are! ($)used "or select operations! reading o" data does not require a instance to dison a globalloc+#

Null (N)allos instances to +eep a loc+ ithout any permission on the bloc+(s)# *his mode is usedso that loc+s need not be created and destroyed all the time! it ust converts "rom one loc+ toanother#

?oc+ roles are used by ache /usion! it can be either local or global! the resource is local i" the bloc+ isdirty only in the local cache! it is global i" the bloc+ is dirty in a remote cache or in several remote caches#

 A -ast ;mage (-;) is +ept by the instance hen a bloc+ is shipped to another instance! the role is thenchanged to a global role! thus the -; represents the state o" a dirty bu""er# A node must +eep a -; until itreceives noti"ication "rom the master that a rite to dis+ has completed covering that version! the node illthen log a bloc+ ritten record (>W<)# ; have already discussed -; and >W< in my bac+up section#

When a ne current bloc+ arrives! the previous -; remains untouched in case another node requires it# ;"

there are a number o" -;0s that exist! they may or may not merge into a single -;! the master illdetermine this based on i" the older -;0s are required! a indeterminate number o" -;0s can exist#

;n the local role only S and I modes are permitted! hen requested by the master instance the holdinginstance serves a copy o" the bloc+ to others# ;" the bloc+ is globally clean this instance loc+ role remainslocal# ;" the bloc+ is modi"ied (dirty)! a -; is retained and the loc+ becomes global# ;n the global loc+ roleloc+ modes can be =! S and I! the bloc+ is global and it may even by dirty in any o" the instances and thedis+ version may be obsolete# ;nterested parties can only modi"y the bloc+ using I mode! an instancecannot read "rom the dis+ as it may not be current! the holding instance can send copies to otherinstances hen instructed by the master#

; have a complete detailed al+though in my cacheD"usion section! hich ill help you better tounderstand#

 A loc+ element holds loc+ state in"ormation (converting! granting! etc)# ?s are managed by the loc+process to determine the mode o" the loc+s! they also old a chain o" cache bu""ers that are covered by the? and allo the .racle database to +eep trac+ o" cache bu""ers that must be ritten to dis+ in a case a? (mode) needs to be dongraded (I N =)#

?s protect all the data bloc+s in the bu""er cache! the list belo describes the classes o" the data bloc+hich are managed by the ?s using CS loc+s (xJbh#class)#

/<

% I?3<

2 SH<3<

$ <4 <A6;=C

' :<.8<,

& ;<.8<,

7 W<;*;=C

-;

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 35/47

So putting this altogether you get the "olloing! CS manages -: loc+s in the C<6! -: loc+smanage the data bloc+s in the global cache# 6ata bloc+s are can be +ept in any o" the instances bu""ercache (hich is global)! i" not "ound then it can be read "rom dis+ by the requesting instance# *he CSmonitors and maintains the list and mode o" the bloc+s in all the instances# ach instance ill master anumber o" resources! but a resource can only be mastered by one instance# CS ensures cachecoherency by requiring that instances acquire a loc+ be"ore modi"ying or reading a database bloc+# CSloc+s are not rolevel loc+s! rolevel loc+s are used in conunction ith -: loc+s# CS loc+ ensuresthat they bloc+ is accessed by one instances then rolevel loc+s manage the bloc+s at the rolevel# ;" abloc+ is modi"ied all -ast ;mages (-;) are no longer current and ne copies are required to obtained#

onsistent read processing means that readers never bloc+ riters! as the same in a single instance#.ne parameter that can help is DdbDbloc+DmaxDcrDdba hich limits the number o" < copies per 6>A onthe bu""er cache# ;" too many < requests arrive "or a particular bu""er! the holder can dison the loc+ onthe bu""er and rite the bu""er to the dis+! thus the requestor can then read it "rom dis+! especially i" therequested bloc+ has a older S= and needs to reconstruct it (+non as < "abrication)# *his istechnically +non as fairness do&nconvert ! and the parameter D"airnessDthreshold can used to con"igureit#

*he lightor+ rule is involved hen < construction involves too much or+ and no current bloc+ or -;

bloc+ is available in the cache "or bloc+ cleanouts# *he belo can be used to vie the number o" times adonconvert occurs

donconvert

select crDrequests! lightDor+s! dataDrequests! "airnessDdonDconverts "romvJcrDbloc+Dserver@

=oteE loer the D"airnessDthreshold i" the ratio goes above 4X! set to i" the instanceis a query only instance#

*he C<6 is a central repository "or loc+s and resources! it is distributed across all nodes (not a singlenode)! but only one instance masters a resource# *he process o" maintaining in"ormation about resourcesis called loc) mastering  or resource mastering # ; spo+e about loc+ remastering in my bac+up section#

<esource a""inity allos the resource mastering o" the "requently used resources on its local node! it usesdynamic resource mastering to move the location o" the resource masters# =ormally resource masteringonly happens hen a instance oins or leaves the <A environment! as o" .racle %g <2 masteringoccurs at the obect level hich helps "inegrained obect remastering# *here are a number o" parametersthat can be used to dynamically remaster an obect

 0+c0affinity0time speci"ies interval minutes "or remastering

 0+c0affinity0limitde"ines the number o" times a instance access the resource be"oreremastering! setting to disable remastering

 0+c0affinity0minimumde"ines the minimum number o" times a instance access the resource be"oreremastering

 0lm0file0affinity disables dynamic remastering "or the obects belonging to those "iles 0lm0!ynamic0remasterin+ enable or disable remastering

,ou should consult .racle be"ore changing any o" the above parameters#

Cac,e #usion

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 36/47

; mentioned above ache /usion in my C<6 section! here ; go into great detail on ho it or+s! ; ill alsoprovide a number o" al+ through examples on my <A system#

ache /usion uses the most e""icient communications as possible to limit the amount o" tra""ic used onthe interconnect! no you don0t need this level o" detail to administer a <A environment but it sure helpsto understand ho <A or+s hen trying to diagnose problems# <A appears to have one large bu""er

but this is not the case! in reality the bu""er caches o" each node remain separate! data bloc+s are sharedthrough distributed loc+ing and messagingoperations# <A copies data bloc+s across the interconnect toother instances as it is more e""icient than reading the dis+! yes memory and netor+ing together are"aster than dis+ ;1.#

5in+

*he trans"er o" a data bloc+ "rom instances bu""er cache to another instances bu""er cache is +no as a ping # As mentioned already hen an instance requires a data bloc+ it sends the request to the loc+master to obtain a loc+ in the desired mode! this process is +non as bloc)ing as$nchronous trap (>AS*)#When an instance receives a >AS* it dongrades the loc+ ASA-! hoever it might have to rite thecorresponding bloc+ to dis+! this operation is +non as dis) ping  or hard ping # 6is+ pings have beenreduce in the later versions o" <A! thus relaying on bloc+ trans"ers more! hoever there ill alays be a

small amount o" dis+ pinging# ;n the neer versions o" <A hen a >AS* is received sending the bloc+or dongrading the loc+ may be de"erred by tens o" milliseconds! this extra time allos the holdinginstance to complete an active transaction and mar+ the bloc+ header appropriately! this ill eliminate anyneed "or the receiving instance to chec+ the status o" the transaction immediately a"ter receiving1reading abloc+# hec+ing the status o" a transaction is an expensive operation that may require access (andpinging) to the related undo segment header and undo data bloc+s as ell# *he parameter

 DgcDde"erDtime can be used to de"ine the duration by hich an instance de"erred dongrading a loc+#

5ast ma+e *locks (5)

;n the C<6 section ; mentioned -ast ;mages (-;s)! basically they are copies o" data bloc+s in the local

bu""er cache o" an instance# When an instance sends a bloc+ it has recently modi"ied to another instance!it preserves a copy o" that bloc+! mar+ing as a -;# *he -; is +ept until that bloc+ is ritten to dis+ by thecurrent oner o" the bloc+# When the bloc+ is ritten to dis+ and is +non to have a global role! indicatingthe presents o" -;s in other instances bu""er caches! CS in"orms the instance holding the -;s to discardthe -;s# When a chec+point is required it in"orms CS o" the rite requirement! CS is responsible "or"inding the most current bloc+ image and in"orming the instance holding that image to per"orm a bloc+rite# CS then in"orms all holders o" the global resource that they can release the bu""ers holding the -;copies o" the bloc+! alloing the global resource to be released# ,ou can vie the past image bloc+spresent in the "ixed table IJ>H

-;sselect state! count(state) "rom IJ>H group by state@=oteE the state column ith is the past images#

Cac,e #usion

ache /usion ; is also +no as consistent read server and as introduced in .racle #%#'! it +eeps a listo" recent transactions that have changed a bloc+#the original data contained in the bloc+ is preserved inthe undo segment! hich can be used to provide consistent read versions o" the bloc+#

;n a single instance the "olloing happens hen reading a bloc+

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 37/47

• When a reader reads a recently modi"ied bloc+! it might "ind an active transaction in the bloc+

• *he reader ill need to read the undo segment header to decide hether the transaction hasbeen committed or not

• ;" the transaction is not committed! the process creates a consistent read (<) version o" thebloc+ in the bu""er cache using the data in the bloc+ and the data stored in the undo segment

• ;" the undo segment shos the transaction is committed! the process has to revisit the bloc+ andclean out the bloc+ (delay bloc+ cleanout) and generate the redo "or the changes#

;n an <A environment i" the process o" reading the bloc+ is on an instance other than the one thatmodi"ied the bloc+! the reader ill have to read the "olloing bloc+s "rom the dis+

• !ata lock to get the data and1or transaction ;6 and 3ndo >yte Address (3>A)

• un!o se+ment ,ea!er lock to "ind the last undo bloc+ used "or the entire transaction

• un!o !ata lock to get the actual record to construct a < image

>e"ore these bloc+s can be read the instance modi"ying the bloc+ ill have to rite those0s bloc+s to dis+!resulting in & ;1. operations# ;n <A the instance can construct a < copy by hope"ully using the abovebloc+s that are still in memory and then sending the < over the interconnect thus reducing & ;1.operations#

RAC roules,ootin+

*his is the one section hat ill be updated "requently as my experience ith <A gros! as <A hasbeen around "or a hile most problems can be resolve ith a simple google loo+up! but a basicunderstanding on here to loo+ "or the problem is required# ;n this section ; ill point you here to loo+ "or problems! every instance in the cluster has its on alert logs! hich is here you ould start to loo+# Alertlogs contain startup and shutdon in"ormation! nodes oining and leaving the cluster! etc#

Here is my complete alert log "ile o" my to node <A starting up#

*he cluster itsel" has a number o" log "iles that can be examined to gain any insight o" occurring problems!the table belo describes the in"ormation that you may need o" the <S components

<ORA0CR$0"O1&crs&lo+ contains trace "iles "or the <S resources

<ORA0CR$0"O1&crs&initcontains trace "iles "or the <S daemon during startup! a good place tostart

<ORA0CR$0"O1&css&lo+contains cluster recon"igurations! missed chec+ins! connects anddisconnects "rom the client SS listener# ?oo+ here to obtain hen rebootsoccur

<ORA0CR$0"O1&css&initcontains core dumps "rom the cluster synchroni9ation service daemon(.Sd)

<ORA0CR$0"O1&em&lo+ log "iles "or the event volume manager and eventlogger daemon

<ORA0CR$0"O1&em&init pid and loc+ "iles "or 8:

<ORA0CR$0"O1&srm&lo+ log "iles "or .racle luster <egistry (.<)

<ORA0CR$0"O1&lo+log "iles "or .racle clusterare hich contains diagnostic messages at the.racle cluster level

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 38/47

 As in a normal .racle single instance environment! a <A environment contains the standard <6>:Slog "iles! these "iles are located by the parameter bac)ground_dest_dump# *he most important o" theseare

<ORAC0*A$&a!min&u!ump contains any trace "ile generated by a user process

<ORAC0*A$&a!min&c!umpcontains core "iles that are generated due to a core dump in a userprocess

=o lets loo+ at a to node startup and the sequence o" events

/irst you must chec+ that the <A environment is using the connect interconnect! this can be done byeither o" the "olloing

log"ileLL *he location o" my alert log! yours may be di""erent1u%1app1oracle1admin1racdb1bdump1alertDracdb%#log

i"c"g command oi"c"g geti"

table chec+ select instDid! pubD+sxpia! pic+edD+sxpia! ipD+sxpia "rom xJ+sxpia@

oradebug

S5?N oradebug setmypidS5?N oradebug ipc

=oteE chec+ the trace "ile hich can be located by the parameter userDdumpDdest

system parameterclusterDinterconnects

=oteE used to speci"y hich address to use

When the instance starts up the ?oc+ :onitor0s (?:.=) ob is to register ith the =ode :onitor (=:) (seebelo table)# <emember hen a node oins or leaves the cluster the C<6 undergoes a recon"igurationevent! as seen in the log"ile it is a seven step process (see belo "or more details on the seven stepprocess)#

*he ?:.= trace "ile also has details about recon"igurations it also details the reason "or the event

reconfi+uationreason

!escription

3means that the =: initiated the recon"iguration event! typical hen a node oins orleaves a cluster 

means that an instance has died

Ho does the <A detect an instance death! every instance updates the control "ileith a heartbeat through its chec+point (F-*)! i" the heartbeat in"ormation ismissing "or x amount o" time! the instance is considered to be dead and the ;nstance:embership <ecovery (;:<) process initiates recon"iguration#

B

means communication "ailure o" a node1s# :essages are sent across theinterconnect i" a message is not received in an amount o" time then acommunication "ailure is assumed by de"ault 36- is used and can be unreliable so+eep an eye on the logs i" too many recon"igurations happen "or reason $#

xample o" arecon"iguration! ta+en"rom the alert log#

Sat :ar 2 %%E$'E'$ 2%Reconfi+uration starte! (ol! inc new inc E)?ist o" nodesE

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 39/47

  %/loal Resource Directory fro7en

  O allocate domain ! invalid K *<3  ommunication channels reestablished  :aster broadcasted resource hash value bitmaps  =onlocal -rocess bloc+s cleaned out

Sat :ar 2 %%E$'E'$ 2%  ?:S E CS shados cancelled! closed  Set master node in"o  Submitted all remoteenqueue requests  6ncvts replayed! 8A?>?Fs dubious  All grantable enqueues granted  -ost S:.= to start %st pass ;<Sat :ar 2 %%E$'E'$ 2%  ?:S E CS shados traversed! $2B% replayedSat :ar 2 %%E$'E'$ 2%  Submitted all CS remotecache requests  -ost S:.= to start %st pass ;<  /ix rite in gcs resources

Reconfi+uration complete

=oteE hen a recon"iguration happens the C<6 is "ro9en until the recon"iguration iscompleted

on"irm that the database has been started in cluster mode! the log "ile ill state the "olloing

cluster mo!eSat :ar 2 %%E$&E2 2%6atabase mounted in Shared :ode (?3S*<D6A*A>ASK*<3)ompletedE A?*< 6A*A>AS :.3=*

Staring ith %g the S= is broadcast across all nodes! the system ill have to ait until all nodes have

seen the commit S=# ,ou can change the board cast method using the system parameter _lg&r_as$nc_broadcasts.

amport Al+orit,m

*he lamport algorithm generates S=s in parallel and they are assigned to transaction on a "irst come"irst served basis! this is di""erent than a single instance environment! a broadcast method is used a"ter acommit operation! this method is more -3 intensive as it has to broadcast the S= "or every commit!but he other nodes can see the committed S= immediately#

*he initiali9ation parameter max_commit_propagation_dela$  limits the maximum delay allo "or S=propagation! by de"ault it is 7 seconds# When set to less than % the broadcast on commit algorithm isused#

Disale&nale Oracle RAC

*here are times hen you may ish to disable <A! this "eature can only be used in a 3nix environment(no indos option)#

Disale Oracle RAC ('ni; only)

%# ?og in as .racle in all nodes

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 40/47

2# shutdon all instances using either normal or immediate option

$# change to the or+ing directory J.<A?DH.:1lib

4. run the belo ma)e command to relin+ the .racle binaries ithout the <A option (should ta+e a"e minutes)

ma+e " insDrdbms#m+ rac0off 

'# =o relin+ the .racle binaries

  ma+e " insDrdbms#m+ ioracle

nale Oracle RAC ('ni; only)

%# ?og in as .racle in all nodes

2# shutdon all instances using either normal or immediate option

$# change to the or+ing directory J.<A?DH.:1lib

4. run the belo ma)e command to relin+ the .racle binaries ithout the <A option (should ta+e a"e minutes)

ma+e " insDrdbms#m+ rac0on

'# =o relin+ the .racle binaries

  ma+e " insDrdbms#m+ ioracle

5erformance ssues

.racle can su""er a number o" di""erent per"ormance problems and can be categori9ed by the "olloing

• Hung 6atabase

• Hung Session(s)

• .verall instance1database per"ormance

• 5uery -er"ormance

 A hung database is basically an internal deadloc+ beteen to processes! usually .racle ill detect thedeadloc+ and rollbac+ one o" the processes! hoever i" the situation occurs ith the internal +ernellevelresources (latches or pins)! it is unable to automatically detect and resolve the deadloc+! thus hanging the

database# When this event occurs you must obtain dumps "rom each o" the instances ($ dumps perinstance in regular times)! the trace "iles ill be very large#

capture in"ormation LL 3sing alter sessionS5?N alter session set maxDdumpD"ileDsi9e K unlimited@S5?N alter session set events 0immediate trace name systemstate level %0@

L using oradebugS5?N select O "rom dual@

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 41/47

S5?N oradebug setmypidS5?N unlimitS5?N oradebug dump systemstate %

L using oradebug "rom another instanceS5?N select O "rom dual@

S5?N oradebug setmypidS5?N unlimitS5?N oradebug g all dump systemstate %

=oteE the select statement above is to avoid problems on pre .racle

S5?-lus problemsconnecting

LL ;" you get problems connecting ith S5?-?3S use the command beloJ sqlplus prelimnter usernameE 1 as sysdba

 A severe per"ormance problem can be mista+en "or a hang! this usually happen because o" contentionproblems! a systemstate dump is normally used to analy9e this problem! hoever a systemstate dumpta+en a long time to complete! it also has a number o" limitations

• <eads the SCA in a dirty manner! so it may be inconsistent

• 3sually dumps a lot o" in"ormation

• does not identi"y interesting processes on hich to per"orm additional dumps

• can be a very expensive operation i" you have a large SCA#

*o overcome these limitations a ne utility command as released ith i called ,an+analy7e hichprovides clusteride in"ormation in a <A environment on a single shot#

sql method alter session set events 0immediate trace hanganaly9e level MlevelN0@

oradebug

S5?N oradebug hanganaly9e MlevelN

LL Another ay using oradebugS5?N setmypidS5?N setinst allS5?N oradebug g de" hanganaly9e MlevelN

=oteE you ill be told here the output ill be dumped to

,an+analy7e leels

3- only hanganaly9e output! no process dump at all! clic+ here "or an example level % dump

B ?evel 2 V 6ump only processes thought to be in a hang (;=DHA=C state)

E ?evel $ V 6ump lea" nodes (bloc+ers) in ait chains (?A/! ?A/D=W! ;C=D6:- state)

4 ?evel 4 V 6ump all processes involved in ait chains (=?A/ state)

32 6ump all processes (;C= state)

*he hanganaly9e command uses internal +ernel calls to determine hether a session is aiting "or aresource and reports the relationship beteen bloc+ers and aiters! systemdump is better but i" you overhelmed try hanganaly9e "irst#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 42/47

Deu++in+ No!e iction

 A node is evicted "rom the cluster a"ter it +ills itsel" because it is not able to service the application! thisgenerally happens hen you have communication problems# /or eviction node problems loo+ "or ora2B74 errors in the alert log "ile and ?:.= trace "iles#

*o understand eviction problems you need to no the basics o" node membership and instancemembership recovery (;:<) or+s# When a communication "ailure happens the heartbeat in"ormation inthe control cannot happen! thus data corruption can happen# ;:< ill remove any nodes "rom the clusterthat it deems as a problem! ;:< ill ensure that the larger part o" the cluster ill survive and +ills anyremaining nodes# ;:< is part o" the service o""ered by luster Croup Services (CS)# ?:.= handlesmany o" the CS "unctionalities! this or+s at the cluster level and can or+ ith $rd party so"tare (Sunluster! 8eritas luster)# *he =ode :onitor (=:) provides in"ormation about nodes and their health byregistering and communicating ith the luster :anager (:)# =ode membership is represented as abitmap in the C<6# ?:.= ill let other nodes +no o" any changes in membership! "or example i" a node

 oins or leaves the cluster! the bitmap is rebuilt and communicated to all nodes#

No!e re+isterin+(alert lo+)

lmon registered ith =: instance id % (internal mem no )

.ne thing to remember is that all nodes must be able to read "rom and rite to the control"ile# CS ma+essure that members are valid! it uses a voting mechanism to chec+ the validity o" each member# ; havealready discussed the voting dis+ in my architecture section! as stated above memberships is held in abitmap in the C<6! the F-* process updates the control"ile every $ seconds in an operation +non as aheartbeat # ;t rites into a single bloc+ that is unique "or each instance! thus intrainstance coordination isnot required! this bloc+ is called the chec)point progress record. ,ou can see the control"ile records usingthe gv!controlfile_record_section vie! all members attempt to obtain a loc+ on the control"ile record "orupdating! the instance that obtains the loc+ tallies the votes "rom all members! the group membershipmust con"orm to the decided (voted) membership be"ore alloing the CS1CS recon"iguration toproceed! the control"ile vote result is stored in the same bloc+ as the heartbeat in the control "ilechec+point progress record#

 A cluster recon"iguration is per"ormed using 7 steps

%# =ame service is "ro9en! the CS contains an internal database o" all the members1instances inthe cluster ith all their con"igurations and servicing details#

2# ?oc+ database (;6?:) is "ro9en! this prevents processes "rom obtaining loc+s on resources thatere mastered by the departing1dead instance

$# 6etermination o" membership and validation and ;:<

4# >itmap rebuild ta+es place! instance name and uniqueness veri"ication! CS must synchroni9ethe cluster to be sure that all members get the recon"iguration event and that they all see thesame bitmap#

'# 6elete all dead instance entries and republish all names nely con"igured

&# 3n"ree9e and release name service "or use

7# Hand over recon"iguration to CS1CS

Deu++in+ CR$ an! /$D

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 43/47

.racle server management con"iguration tools include a diagnostic and tracing "acility "or verbose output"or S<8*?! CS6! CS6*? or S<8.=/;C#

*o capture diagnose "olloing the belo

%# use vi to edit the gsd#sh1srvctl1srvcon"ig "ile in the J.<A?DH.:1bin directory

2# At the end o" the "ile loo+ "or the belo line

exec JG< classpath J?ASS-A*H oracle#ops#mgmt#daemon#.-S:6aemon J:,D.H.:

$# Add the "olloing ust be"ore the classpath in the exec JG< line

6*<A;=C#=A>?6Ktrue 6*<A;=C#?8?K2

4# the string should loo+ li+e this

exec JG< 6*<A;=C#=A>?6Ktrue 6*<A;=C#?8?K2 classpath###########

;n .racle database %g setting the belo variable accomplishes the same thing! set it to blan+ to remove

the debugging

nable tracing J export S<8:D*<AKtrue

6isable tracing J export S<8:D*<AKUU

 Adding or 6eleting a =ode

.ne o" the obs o" a 6>A is adding and removing nodes "rom a <A environment hen capacitydemands! although you should add a node o" a similar spec it is possible to add a node o" a higher orloer spec#

*he "irst stage is to con"igure the operating system and ma+e sure any necessary drivers are installed!also ma+e sure that the node can see the shared dis+s available to the existing <A#

; am going to presume e have a to <A environment already setup! and e are going to add a thirdnode#

5re-nstall C,eckin+

,ou used the luster 8eri"ication utility hen installing the <A environment! the tools chec+ that thenode has been properly prepared "or a <A deployment# ,ou can run the command either "rom the nenode or "rom any o" the existing nodes in the cluster 

preinstall chec+ run "romne node

runcluv"y#sh stage pre crsinst n rac%!rac2!rac$ r %gr2

preinstall chec+ run "romexisting node

cluv"y stage pre crsinst n rac%!rac2!rac$ r %g2

:a+e sure that you "ix any highlighted problems be"ore continuing#

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 44/47

nstall CR$

luster <eady Services (<S) should be installed "irst! this allos the node to become part o" the cluster# Adding the ne node can be started "rom any o" the existing nodes

%# ?og into any o" the existing nodes as user oracle then run the belo command! the script belo

starts the .3; C3; tool! hope"ully the tool ill already see the existing cluster and ill "ill in thedetails "or you

J.<AD<SDH.:1oui1bin1addnode#sh2# ;n the specify cluster no!es to a!! to installation screen! enter the ne names "or the public!

private and virtual hosts

$# lic+ ne;t to see a summary page

4# lic+ install! the installer ill copy the "iles "rom the existing node to the ne node# .nce copiedyou ill be as+ed to run orainstRoot.s, and root.s, as user root

'# <un orainstRoot.s, and root.s, in the ne and roota!!no!e.s, in the node that you arerunning the installation "rom#

 

orainstRoot.s,sets the .racle inventory in the ne node and set onerships and permissions to theinventory

root.s,chec+s hether the .racle <S stac+ is already con"igured in the ne node! creates1etc1oracle directory and adds the relevant .< +eys to the cluster registry and it addsthe daemon to <S and starts <S in the ne node#

roota!!no!e.s, con"igures the .< registry to include the ne nodes as part o" the cluster

&#

7# lic+ ne;t to complete the installation# =o you need to con"igure .racle =oti"ication Services

(.=S)# *he port can be identi"ied by the belo command

  cat J.<AD<SDH.:1opmn1con"1ons#con"ig

# =o run the .=S utility by supplying the MremoteDportN number obtained above

  racgons addDcon"ig rac$EMremoteDportN

nstallin+ Oracle D* $oftware

.nce the <S has been installed and the ne node is in the cluster! it is time to install the .racle 6>so"tare# Again you can use any o" the existing nodes to install the so"tare#

%# ?og into any o" the existing nodes as user oracle then run the belo command! the script belostarts the .3; C3; tool! hope"ully the tool ill already see the existing cluster and "ill in the details"or you

J.<AD<SDH.:1oui1bin1addnode#sh2# lic+ ne;t on the elcome screen to open the specify cluster no!es to a!! to installation 

screen! you should have a list o" all the existing nodes in the cluster! select the ne node andclic+ ne;t

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 45/47

$# hec+ the summary page then clic+ install to start the installation

4# *he "iles ill be copied to the ne node! the script ill as+ you to run run.s, on the ne node!then clic+ O to "inish o"" the installation

Confi+urin+ t,e istener

=o its time to con"igure the listener in the ne node

%# ?ogin as user oracle! and set your D$5AF environment variable! then start the =etor+on"iguration Assistant

J.<A?DH.:1bin1netca2# hoose cluster mana+ement

$# hoose listener 

4# hoose a!!

'# hoose the the name as ?;S*=<

*hese steps ill add a listener on rac$ as ?;S*=<Drac$

Create t,e Dataase nstance

<un the belo to create the database instance on the ne node

%# ?ogin as oracle on the ne node! set the environment to database home and then run thedatabase creation assistant (6>A)

J.<A?DH.:1bin1dbca2# ;n the elcome screen choose oracle real application clusters !ataase to create the instance

and clic+ ne;t

$# hoose instance mana+ement and clic+ ne;t

4# hoose a!! instance and clic+ ne;t

'# Select RACD* (or hatever name you gave you <A environment) as the database and enterthe S,S6>A and passord! clic+ ne;t

&# ,ou should see a list o" existing instances! clic+ ne;t and on the "olloing screen enter.<A<A$ as the instance and choose <A$ as the node name (substitute any o" the abovenames "or your environment naming convention)

7# *he database instance ill no created! clic+ next in the !ataase stora+e screen#! choose yes

hen as+ed to extend AS:

Remoin+ a No!e

<emoving a node is similar to above but in reverse order 

%# 6elete the instance on the node to be removed2# lean up AS:

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 46/47

$# <emove the listener "rom the node to be removed

4# <emove the node "rom the database

'# <emove the node "rom the clusterare

,ou can delete the instance by using the database creation assistant (6>A)! invo+e the program choosethe <A database! choose instance mana+ement and then choose !elete instance! enter the sysdbauser and passord then choose the instance to delete#

*o clean up AS: "ollo the belo steps

%# /rom node % run the belo command to stop AS: on the node to be removed

  srvctl stop asm n rac$  srvctl remove asm n rac$

2# =o run the "olloing on the node to be removed

  cd J.<A?DH.:1admin

  rm r" VAS: cd J.<A?DH.:1dbs

  rm " OAS:O

$# hec+ that 1etc1oratab "ile has no AS: entries! i" so remove them

=o remove the listener "or the node to be removed

%# ?ogin as user oracle! and set your D$5AF environment variable! then start the =etor+on"iguration Assistant

J.<A?DH.:1bin1netca

2# hoose cluster mana+ement

$# hoose listener 

4# hoose Remoe

'# hoose the the name as ?;S*=<

=ext e remove the node "rom the database

%# <un the belo script "rom the node to be removed

cd J.<A?DH.:1bin#1run;nstaller update=ode?ist .<A?DH.:KJ.<A?DH.: U?3S*<D=.6SKYrac$ZUlocal#1run;nstaller 

2# hoose to deinstall products and select the dbhome

$# <un the "olloing "rom node %

  cd J.<A?DH.:1oui1bin  #1run;nstaller update=ode?ist .<A?DH.:KJ.<A?DH.:U?3S*<D=.6SKYrac%!rac2!rac$ZU

7/18/2019 RAC NOTES_p

http://slidepdf.com/reader/full/rac-notesp 47/47

?astly e remove the clusterare so"tare

%# <un the "olloing "rom node %! you obtain the port number "rom remoteport section in theons#con"ig "ile in J.<AD<SDH.:1opmn1con"

J<SDH.:1bin1racgons removeDcon"ig rac$E&2

2# <un the "olloing "rom the node to be removed as user root

  cd J<SDH.:1install  #1rootdelete#sh

$# =o run the "olloing "rom node % as user root! obtain the node number "irst

J<SDH.:1bin1olsnodes n  cd J<SDH.:1install  #1rootdeletenode#sh rac$!$

4# =o run the belo "rom the node to be removed as user oracle

  cd J<SDH.:1oui1bin

  #1run;nstaller update=ode?ist .<A?DH.:KJ.<A?DH.:U?3S*<D=.6SKYrac$ZU <SK*<3 local  #1run;nstaller 

'# hoose to deinstall so"tare and remove the <SDH.:

&# <un the "olloing "rom node as user oracle

  cd J<SDH.:1oui1bin#1run;nstaller update=ode?ist .<A?DH.:KJ.<A?DH.:

U?3S*<D=.6SKYrac%!rac2!rac$ZU <SK*<3

7# hec+ that the node has been removed! the "irst should report Uinvalid nodeU! the second youshould not see any output and the last command you should only see nodes rac% and rac2

srvctl status nodeapps n rac$  crsDstat Qgrep i rac$  olsnodes n