Oracle RAC 12c and Policy-Managed Databases, a Technical Overview

COLLABORATE 14 – IOUG Forum

Database

1 | P a g e “ORACL E 12C DATA GU AR D: FAR SYNC, R EAL-T IME CASCAD E ST ANDBY AN D OTHER GOODI ES”

Whit e Pa pe r

Oracle RAC 12c and Policy-Managed Databases, a Technical Overview

Ludovico Caldara, Trivadis AG

ABSTRACT

Oracle RAC Policy-Managed Database (PMD) is a powerful but so far rarely used feature introduced in Oracle Database 11g Release 2 to automate the instance administration in a dynamic, multi-node cluster.

The aim of this presentation is to review how PMD works, how to implement and administer it successfully, and how to benefit from this technology compared to the traditional administrator-managed deployment. During the session, the new features of the 12c Grid Infrastructure related to PMD are highlighted.

TARGET AUDIENCE

This whitepaper is targeted at database administrators and architects who are familiar with Oracle Real Application Cluster concepts and want to know more about Policy-Managed Databases.

EXECUTIVE SUMMARY After reading this white-paper, you should be able to:

• Evaluate the features of PMD

• Describe how to plan and administer the solution

• Identify the reasons to migrate to a PMD deployment


Database

2 | P a g e “ORACL E 12C DATA GUARD: FAR SYN C, REAL-T IME CASCADE STANDBY AND OTHER GOODI ES”

Whit e Pa pe r

ADMIN VS POLICY MANAGEMENT Oracle RAC admin-managed databases are so far the most common type of configuration used by Oracle customers to administer databases, probably because the behavior is unchanged from Oracle 10g onwards and the DBAs feel more confident with it. Admin-Managed databases provide high performance, high availability and scalability; however the process of creating and configuring new instances is not completely straightforward: DBAs have to take care of creating the directory structures, create the undo tablespaces dedicated to the instances, create the new redo threads and configure the instances to be managed by the cluster:

srvctl add instance –db dbname –instance sid –node server There is a strong relationship between the instance and the server, so the DBAs must plan which instances will run on which servers and prepare everything in advance; but where new nodes are added to the cluster, things become even more complex.

Policy-Managed databases don’t have preferred servers. The databases are rather assigned to a pool of servers whose size can change dynamically.

$ srvctl modify serverpool –serverpool hr –min 4

Instead of adding new instances to the database, the DBAs can just expand the server pool in order to have the database to expand with it. The instance configuration is totally delegated to the cluster that will automatically add UNDO tablespaces and redo logs as needed (provided that ASM or Oracle-Managed Files are in place), allowing having clusters that can grow or shrink dynamically. This new behavior is an enabler for ease of consolidation and rapid provisioning, thus making a step forward in the direction of elastic database clouds.

Another important improvement introduced by policy-managed databases resides in the resiliency of the clusters.

Imagine that you have two different databases, HR and CRM, the latter with higher importance in your company. If you lose one node belonging to CRM database, with admin-managed databases you should do a manual intervention: stop one HR instance and start another CRM instance, provided that it has been previously configured to run on the surviving server. If it’s not the case, configuring from scratch the new instance can be time consuming and error prone, which is not recommended during a resource shortage.


Database


Whit e Pa pe r

Policy-Managed Databases solve this kind of problem automatically: DBAs can assign priorities to server pools and let them resize automatically in order to meet resource requirements of most important databases.

There’s no requirement for an immediate DBA intervention (well, someone should at least solve the node failure!): there’s a higher level of availability for the higher prioritized databases.

SERVER POOLS Server pools are the foundation of policy managed databases. They are used to partition a cluster into smaller groups of servers. Because databases are assigned to server pools, their size directly affects the resources available to the databases.

There are always two predefined server pools. The “Free” server pool consists of new or unassigned nodes, whereas the “Generic” server pool consists of servers running admin-managed databases.

Policy-managed databases cannot be assigned to the Generic server pool and the instances of admin-managed databases cannot run in server pools other than the Generic. Running policy-managed databases requires then the creation of additional server pools.


Database


Whit e Pa pe r

You can have both admin-managed and policy-managed databases into the same cluster. This may be necessary in order to consolidate pre-11gR2 admin-managed databases that cannot be policy-managed.

The conversion from admin-managed to policy-managed databases is achieved by:

• Having an existing user-defined server pool, or creating a new one:

$ srvctl add srvpool -serverpool hr -min 1 -max 2 - importance 10

• Moving the database to the server pool:

$ srvctl modify database –db hr -serverpool hr The clusterware will stop the database and restart it as policy-managed. Sadly, there’s no way to convert it without service disruption. If you have just two nodes (e.g. you want to convert an Oracle Database Appliance to policy-managed), you can migrate by following these steps:

• free up a node for usage in the new pool by stopping one instance and removing it from the cluster (srvctl remove instance)

• create the server pool

• move the database to the newly created server pool (db is converted to policy-managed)

• remove the last admin-managed instance from the cluster (that should be done automatically)

SERVER ASSIGNMENT

The size of server pools is driven by three main attributes:

• MIN_SIZE: The minimum number of servers that compose the server pool

• MAX_SIZE: The maximum number of servers

• IMPORTANCE: The priority level (from 0 to 1000, higher numbers means higher priority)

The cluster uses these attributes when it has to decide for server assignment. Free and Generic server pools have by default all attributes set to 0.

When the cluster starts:

• The cluster starts by assigning servers to the most important server pools, up to MIN_SIZE:


Database


Whit e Pa pe r

• Once the first servers are available to the server pool, the corresponding databases are also started. This allows

DBAs to start most important databases first.

• Then, in descending order of importance, the other server pool minimum sizes are fulfilled

• When all minimum sizes are met, the same is done for the maximum sizes.

• When all maximum sizes are met, eventual spare servers will belong to the “Free” server pool.

When a node fails or is removed from the cluster:

• If a node fails in a server pool, but its size is still greater than its MIN_SIZE attribute, no action is taken, no matter what the importance is.

• If a node fails in a server pool causing its size to fall below MIN_SIZE, the cluster draws a server from a server pool with more than MIN_SIZE servers, in ascending order of importance. This can actually be a server pool with an importance higher than the failed server pool, because the MIN_SIZE will be still fulfilled.

• If a node fails in a server pool and its size falls below MIN_SIZE but all server pool sizes are already equal or

below their respective MIN_SIZE, the server is taken from the pool with the lowest importance. This can lead to having server pools with low importance and no available servers (thus, offline databases) and server pools with higher importance with more than one server available. This is ok, but it is important that you consider it when configuring your parameters.


Database


Whit e Pa pe r

When a node is added or a server pool is modified, the cluster will recalculate and reallocate servers in order to satisfy the requirements, using the same rules described above.

When a cluster is upgraded, the upgrade process will move resources gracefully and automatically in order to satisfy all the requirements. E.g., the upgrade of a cluster from 11gR2 to 12c will let you organize the execution of the root.sh script in batches that will stop and upgrade up to three nodes at time. With policy-managed databases, it doesn’t matter how you group servers in batches, because when the CRS is stopped on the nodes, the server pools size change depending on the rules defined by the DBAs. This greatly increases the online upgrade capabilities of the cluster.

WHAT-IF CONDITIONS

Oracle Database 12c comes with a new feature called “What-If” that is handy to evaluate the result of a command without actually executing it. You should use it whenever you have to modify server pool attributes and you’re unsure of how your services and databases will react.

When using crsctl, there is an “eval” command that triggers the evaluation:

$ crsctl eval modify serverpool ora.crm -attr "MIN_SIZE=4,MAX_SIZ E=4" –f […] Resource 'ora.hr.payroll.svc' (1/1) will be in state [OFFLINE] Server 'rac4' will be moved from pools [ora.hr] to pools [ora.crm] Resource 'ora.hr.db' (2/1) will be in state [OFFLIN E]

When using srvctl, you can use the “-eval” flag:

$ srvctl add serverpool -serverpool bi -min 5 -max 5 -importance 10 –eval Service payroll will be stopped on node rac4 Database hr will be stopped on node rac4 Server rac4 will be moved from pool hr to pool bi

NESTED SERVER POOLS

Server pools can be nested; you can create “child pools” with crsctl by setting the attribute “PARENT_POOLS”. Child pools creation using srvctl is not possible.

However, using nested server pools in conjunction with policy-managed database doesn’t work as expected and it’s not fully documented by Oracle. Because of that I do not recommend their usage.


Database

7 | P a g e “ORACL E 12C DATA GU AR D: FAR SYNC, REAL-T IME CASCADE ST ANDBY AND OTHER GOODI ES”

Whit e Pa pe r

POLICY SETS

Sometimes Oracle customers need to have different server pool configurations at different times, depending on how workload changes over time. As an example, a customer may want to have more servers (thus more resources) for the server pool HR once a month in order to process payrolls and more servers for the server pool DWH for nightly reports.

Prior to version 12c, the only way to change the workload to fit the requirements was to execute several “srvctl modify serverpool”. Oracle Database 12c has introduced the policy sets, a feature that allows preparing in advance all the different workload scenarios and activating them with a single command.

The easiest way to create policy sets is dumping the current configuration into a file, edit it, and reload it into the cluster.

The following command will create a file containing the current configuration:

$ crsctl create policyset -file cluster_policies.tx t

This is an example of content:

SERVER_POOL_NAMES=Free ora.dwh ora.hr POLICY NAME=Default SERVERPOOL NAME=ora.dwh IMPORTANCE=5 MAX_SIZE=2 MIN_SIZE=1 SERVER_CATEGORY=ora.hub.category SERVERPOOL NAME=ora.hr IMPORTANCE=8 MAX_SIZE=4 MIN_SIZE=3 SERVER_CATEGORY=ora.hub.category You can append new policies to the files. As an example:

POLICY NAME=monthly_payroll SERVERPOOL NAME=ora.hr IMPORTANCE=10 MIN_SIZE=3 MAX_SIZE=3 SERVERPOOL NAME=ora.dwh IMPORTANCE=5 MIN_SIZE=1 MAX_SIZE=1 POLICY NAME=nightly_reports SERVERPOOL


Database


Whit e Pa pe r

NAME=ora.hr IMPORTANCE=5 MIN_SIZE=1 MAX_SIZE=1 SERVERPOOL NAME=ora.dwh IMPORTANCE=5 MIN_SIZE=3 MAX_SIZE=4 Then you must reload the file by using the “modify policyset” command:

$ crsctl modify policyset -file cluster_policies.tx t The new policies can be activated by modifying the “LAST_ACTIVATED_POLICY” attribute of the policyset:

$ crsctl modify policyset –attr "LAST_ACTIVATED_POL ICY=monthly_payroll"

The command “crsctl eval modify policyset” is not implemented yet (as of release 12.1.0.1.2), you cannot benefit from it if you want to do a what-if analysis prior to switch the active policy.

There’s no way to switch policies automatically from the Oracle side, policies are not integrated in the resource manager, you have to script the switch and run it from a scheduler. The same kind of functionality however comes with the Quality of Services Management product that complements policy-managed databases and provides inter-database resource management.

DATABASE BEHAVIOR With admin-managed databases, as we’ve seen, you must set manually which instances run on which servers. This is somehow an additional effort on one side, but on the other it allows DBAs to choose how many instances can run for each database.

Policy-managed databases however span the entire server pool, so the number of instances is always equal to the number of servers composing the server pool, and it is the same for each database in the server pool.


Database


Whit e Pa pe r

The total number of instances is the result of (#DATABASES * #(SERVERS in the pool)). This can lead to instance proliferation and the physical memory of each server can limit the number of databases running in the server pool independently from the pool size. To work around this problem, you can rely on RAC One Node feature for less critical databases, this can dramatically increase the number of running databases but the overcome is decreased high availability:

I recommend having a mix of both RAC and RAC One Node databases in the cluster in order to host both critical and non-critical databases.

When consolidating many database instances on the same server, make sure you use instance caging (CPU_COUNT) to prevent one instance from consuming all the CPU of the server. In order to manage also other kind of resources, Oracle recommends using Quality of Services management feature. Although QOS is quite complex and almost unfamiliar, it’s worth to consider it for big consolidation projects.

Another viable way to reduce the instance footprint and manage resources between databases is to implement Oracle Multitenant. The resources available to the different PDBs can be managed through the new PDB share management of Resource Manager 12c.


Database

10 | P a g e “ORACL E 12C DATA GU AR D: FAR SYNC, R EAL-T IME CASCAD E ST ANDBY AN D OTHER GOODI ES”

Whit e Pa pe r

SERVICE BEHAVIOR Like the databases, policy-managed services can either span the entire server pool or run on a single instance. You control the behavior by specifying either UNIFORM or SINGLETON cardinality:

$ srvctl add service –db CRM –serverpool mixed –ser vice REPORTING –cardinality SINGLETON

$ srvctl add service –db CRM –serverpool mixed –ser vice FIN_READ –cardinality SINGLETON

$ srvctl add service –db CRM –serverpool mixed –ser vice SALES –cardinality UNIFORM

There is no way to predict on which instance a SINGLETON service will run and it’s no more possible to specify the preferred instance upon the service creation.

PRACTICAL TIPS

DON’T MONITOR INSTANCES WITH CRSCTL

With policy-managed databases, the size of server pools and databases can expand or shrink depending on several factors, so it is expected that in some situations instances shut down to honor a server pool shrink. After an instance shuts down, however, the command crsctl will still show it, inducing the DBA to think that an instance is unexpectedly offline:

# crsctl status resource ora.CRM.db -t --------------------------------------------------- ------------------- Name Target State Server State details --------------------------------------------------- ------------------- ora.CRM.db 1 ONLINE ONLINE rac4 Open,STABLE 2 ONLINE ONLINE rac3 Open,STABLE 3 ONLINE OFFLINE STABLE If you need to monitor the database instances status, you should rely rather on srvctl and count the number of instances currently running and expect they don’t fall below the MIN_SIZE you have set.:

# srvctl status database -db CRM Instance CRM_1 is running on node node4 Instance CRM_2 is running on node node3

CENTRALIZE THE DIAGNOSTIC_DEST

Server maintenance, cluster restarts and other activities will make instances moving around and change often the server they are running on. If you rely on local filesystems for your diagnostic_dest you may not be able to find which alert file on which server actually contains the log information you need.


Database

11 | P a g e “ORACLE 12C D AT A G UAR D: FAR SYNC, R EAL-T IME CASCADE ST ANDBY AND OTHER GOODI ES”

Whit e Pa pe r

It’s a good idea to point the diagnostic_dest to a common filesystem, ideally ACFS if it’s supported by your platform. Don’t forget to register the ACFS filesystem into the cluster configuration as a dependency of your databases.

MY CONCLUSION Policy-management is an interesting piece of software. It’s more resistant to server failures than the traditional admin-managed databases; it allows very fast instance provisioning and has a proven agility and scaling. The administration becomes easier and many tasks like upgrades and migrations are easier to accomplish. As of 12.1.0.1.1, policy-management is also a requirement for Quality of Services management and Global Data Services. As a general suggestion, I recommend it for every RAC-based database consolidation project.

There are however some fallbacks that require attention. The limited cardinality choice of services and databases (UNIFORM or SINGLETON) doesn’t allow manual, strategic instance placement. The Enterprise Manager, and the crsctl command itself, are not always as reactive as one would expect when new instances appear, relocate or disappear. Some features don’t work together, in particular child pools and resource customization with crsctl may cause configuration errors that are sometimes difficult to solve without service disruption.

REFERENCES

• Oracle® Clusterware Administration and Deployment Guide 12c Release 1 (12.1)

http://docs.oracle.com/cd/E16655_01/rac.121/e17886/toc.htm

• Oracle® Real Application Clusters Administration and Deployment Guide 12c Release 1 (12.1)


• Oracle® Database 2 Day + Real Application Clusters Guide 12c Release 1 (12.1)


Oracle RAC 12c and Policy-Managed Databases, a Technical Overview

Technology

Transcript of Oracle RAC 12c and Policy-Managed Databases, a Technical Overview