DESIGN AND ARCHITECTURE OF ORACLE RAC/ASM CLUSTERED …€¦ · approach for a multi-cluster...

DESIGN AND ARCHITECTURE OF ORACLE RAC/ASM CLUSTERED SOLUTIONArmando RodriguezAdvisory Systems Engineer, EMC [email protected]

2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Business Challenge......................................................................................................... 3

Design Philosophy........................................................................................................... 3

Building Block Description ............................................................................................... 3

VNX MCx multi-core technology Benefits ........................................................................ 4

VNX 8000 Building Block ................................................................................................ 5

Storage Grid Design utilizing VNX MCx 8000 Building Blocks ........................................ 5

Seven important design considerations for the Storage Grid .......................................... 7

Manage efficiently by leveraging EMC Storage software tools ........................................ 9

Conclusion .................................................................................................................... 13

References .................................................................................................................... 14

Disclaimer: The views, processes, or methodologies published in this article are those of the

author. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.


Business Challenge

Demand for high performance, high capacity, workload-optimized data warehouse and analytics

solutions have never been higher. The ability to extract information from today’s large scale data

warehouse environment requires a flexible, highly scalable storage solution that provides

demonstrable return on investment required in today’s competitive environment. EMC’s VNX®

MCx Multi-Core platform provides a balance of performance, capacity, capital cost, and

operating cost that is unrivaled in the industry. This article will explore a scale out, building block

approach for a multi-cluster Storage Grid architecture utilizing VNX with an Oracle RAC (Real

Application Clusters) data warehouse solution.

Design Philosophy

The Storage Grid concept design philosophy is rooted in a few basic principles.

Achieve high levels of scale out performance or capacity by linking low cost, high

performance storage arrays together in a single turnkey solution.

Utilize low cost commodity-based components in all layers of the design and include

only the minimal required software to operate the solution.

The solution specifications should have limited variations to allow for repeatable build

processes in order to minimize time and services required to deploy. A building block

approach is the preferred method to approach the design.

Taken as a whole, the design philosophy allows for a low cost scalable storage solution to meet

a variety of Oracle RAC ASM (Automatic Storage Management) use cases. This includes

everything from high performance OLTP transactional systems to multi-petabyte data

warehouse applications. The focus of this article will be on the data warehouse use case.

Building Block Description

The EMC VNX MCx storage line includes multiple models and sizes to meet different customer

requirements. The entry-level VNXe includes both block and file virtualized into a dual controller

system. The general purpose VNX arrays start with a 250-drive model—VNX5400 —through the

multiprocessor VNX8000 system that scales to 1500 drives. The VNX8000 is the array selected

as the base building block for the scale-out Storage Grid. The rationale for this is three-fold.

1. The array scalability allows the solution to minimize the number of arrays per Storage

Grid required, easing operational management of the solution. Reducing the number of


widget one has to manage improves the administration overhead associated with large

scale out storage farms.

2. The large amount of system cache offered by this particular model allows, 256GB

(Gigabytes) per array allows for improved efficiency in managing the IO workload in

particular write bursts into the system.

3. The VNX8000 includes a powerful Intel-based Dual Socket 8 Core 2.7GHz processor

which in conjunction with the VNX multithreaded code, enables maximum scalability.

Each array has a total of 32 Intel processor cores, making it an ideal candidate as a building

block of a high bandwidth-based Oracle RAC data warehouse cluster.

VNX MCx multi-core technology Benefits

VNX MCx Multi-core technology makes the VNX platform an ideal candidate for a scale out

storage data warehouse application. The performance-optimized multithreaded technology

allows for linear I/O performance scalability utilizing all processor core available in the storage

system. Figure 1 depicts the CPU core balance utilization of each of the 16 Intel cores on the

VNX8000 platform under high bandwidth Oracle RAC I/O load, which peaks at 18GB/s in this

example.

Figure 1: CPU Core Utilization under heavy I/O bandwidth (balance I/O against all Intel cores)


VNX 8000 Building Block

VNX MCx 8000 Array

Dual Socket 8 Core 2.7Ghz Intel Processors

2-Rack Design utilizing 2.5” drives

o Balance floor space efficiency and power

o Power envelope of 2 x 30A circuits per rack

900 2.5” Drive slots per Building Block

256 GB System Memory per Array

Dual redundant system architecture

No single point of failure

Up to 25GB/s+ large block Read Bandwidth

Lithium ION Battery-backed Cache

Figure 2: VNX8000 2 Rack Design

The building block outlined in Figure 2 includes the following characteristics. Each array is

composed of 900 2.5” drive slots per building block. The VNX has many drive options available

that include 7200 RPM SATA, 10k RPM SAS, 15k RPM SAS, and Enterprise Flash Drive (EFD)

in the 2.5” form factor. All drive types are available in multiple capacity amounts and can be

chosen to optimize the capacity / performance requirements for the specific workload profile

being targeted by the design. If targeting a high capacity large scale data warehouse

application, the 900GB 10k 2.5” drive is an ideal candidate that balances performance, capacity,

density, and cost. Additional drive options can be utilized depending on unique application I/O

requirements. Factors to consider when selecting drive types include workload skew, I/O size,

read/write mix, capacity requirements, and cost constraints. Your EMC sales performance

experts can help select the right drive types for your unique requirements.

Storage Grid Design utilizing VNX MCx 8000 Building Blocks

Now, let’s compose an example solution. The requirements are for a high bandwidth Oracle

RAC data warehouse cluster using ASM to stripe the entire workload across all storage

resources. Oracle ASM enables a database to spread workload across multiple LUNs. This

software load balancing technique allows a database administrator to tie together multiple


storage resources on a single array or multiple arrays to achieve very high performance. The

bandwidth design targets in the example below are for 100GB/s composed of 95% large block

reads (256kb+). The Storage Grid is composed of 6 VNX 8000 arrays each with 900 2.5” drive

slots (5400 drives total) with three 16-node Oracle RAC clusters. Host storage Fiber Channel

connectivity is designed to allow for bursts up to 6 GB/s per node based on available HBA

bandwidth utilizing 4 x 16Gb connections per host.


Seven important design considerations for the Storage Grid

1. Balance Fiber Channel connectivity to match performance capability of the storage array

and hosts.

To maximize the build, cost, and support efficiency of the solution, do not over-build the fiber

channel connectivity of the solution. Consider the storage bandwidth capabilities of the

nodes and storage platform when making the selection. The VNX8000 is capable of

providing 72 8Gb/s fiber channel connections per array. However, 32-48 8Gb/s fiber channel

connections are more in line with the platform bandwidth capabilities and strike a nice

balance between performance and cost. 600 MB/s per 8Gb connection is a good guideline

for a high bandwidth read-focused design on the VNX 8000. Since host bandwidth

capabilities depend heavily on the number of cores available to Oracle for I/O processing, a

good rule of thumb for the host is to build to 150MB/s peak per processor core of read

bandwidth. Thus, if you have a quad socket (8 core per socket) host, you should build the

Fiber Channel connectivity to 150MB/s * 8 * 4 = 4800 MB/s per node. Four 16Gb/s fiber

channel connections per host strikes a good balance between bandwidth, cost and cabling

complexity for a typical Oracle RAC data warehouse Storage Grid deployment utilizing quad

processor socket hosts.

2. Even the load distribution between all storage array resources (storage processors, disks,

buses, front end connections).

An important design consideration for Oracle RAC and ASM is to ensure even distribution of

all storage array resources to each ASM disk group. Balancing all resources such as

storage processors, Fiber Channel connectivity, LUN paths, backend storage bus layout,

and all disks is critical to achieve maximum scalability. All clusters should touch all disks in

the solution. This way, the ebb and flow of a given cluster/host workload can leverage all

available storage processing resources. Balancing storage I/O processing resources over

time will maximize the number of Oracle loads and extracts one can achieve in a given day.

3. Assess criteria in selecting Pools or classic RAID group storage layout to be utilized with the

building blocks.

Pools versus classic RAID group storage are an important design consideration for the

layout of the storage arrays. The key consideration is the amount of writes the solution is


expected to have during production workloads. A classic RAID should be considered if a lot

of logging activity and loads occur in the data warehouse on a regular basis. Meanwhile,

Pools provide more flexibility and ease of use but at a cost to in write performance. Writes

coming into the system with Pools both have to commit the writes with the associated RAID

protection overhead and also conduct meta data updates. A heavy write environment favors

classic RAID group over Pools to maximize performance. Pools should always be thickly

provisioned (full allocation) for high bandwidth read Oracle RAC data warehouse use cases.

4. Utilize Brocade or Cisco edge switches as Fibre Channel connectivity mechanisms to

maximize the solution’s cost efficiency. Leverage 16GB/s connectivity wherever possible for

data warehouse high bandwidth application use cases.

The fixed configuration Storage Grid design favors simplicity and cost efficiency. The

selection of low cost, high port density edge switches is ideal for Fibre Channel connectivity.

Selecting low cost edge switches will provide all the availability and performance required by

the Storage Grid. A minimum of two edge switches per Storage Grid is recommended, with

four being the ideal Fibre Channel fabric design. No inter switch links (ISL’s) are needed on

the Fibre Channel switches. Each switch operates on its own unique fabric with no port over

subscription to maximize performance and availability of the Storage Grid.

5. Utilize high speed, low latency network interconnect between host nodes. Two typical

examples are Infiniband or low latency, hardware accelerated TCP.

To provide reliable and efficient scalability of the host Oracle RAC cluster, it is best practice

to utilize a low latency network interconnect between the host nodes. The larger the cluster,

the more a low latency interconnect is required. Infiniband is a common interconnect

technology that provides reliable, low latency and high bandwidth capabilities which are

ideal for the node interconnect in a large scale multimode Oracle RAC cluster.

6. Utilize two or three Oracle RAC Clusters behind each Storage Grid to maximize disk

utilization over time.

To maximize storage bandwidth use over time, using multiple clusters enables the natural

ebb and flow of job processing to share a common storage bandwidth pipe. The goal of

having multiple clusters is to smooth out bandwidth utilization during daily, weekly, and

monthly cluster job loads. The primary drawback of multiple clusters is higher seek


distances on the individual disk drives supporting the solution. However, the bandwidth

utilization gains will typically outweigh this drawback by maximizing capital expenditures of

the solution over its life span.

7. Leveraging management tools and fault alert automation to efficiently support each Storage

Grid is critical.

A VNX scale-out model can cause the number of widgets to manage to grow exponentially.

It is important that automation is used to monitor for key performance indicators and

hardware faults on the storage solution. Knowing when response time volatility is being

observed or when component failures occur will provide early warning and narrow the focus

of the issue to allow for speedier resolution. Alert automation enables the user to narrow the

focus of troubleshooting efforts to specific arrays and specific LUNs that may be having

performance issues. Absent robust monitoring tools, a user is left hunting in the dark to

address performance and or availability issues that are affecting the Oracle RAC database

instance. Three key management tools are available for the VNX to address these

challenges:

1. Unisphere® Remote

2. VNX Monitoring and Reporting

3. EMC’s Secure Remote Support Gateway

Manage efficiently by leveraging EMC Storage software tools

Unisphere Remote

Unisphere Remote, a management tool included with a VNX MCx array that is packaged as a

VMware vAPP (pre-packaged application stack), is able to manage up to 1000 individual VNX

arrays via a single GUI. The scalability limits increase with each generation of the product.

Unisphere Remote monitors health and alerts for hardware faults across the entire VNX

footprint on a given network. Each individual VNX can be tagged into unique groups to enable

viewing system health state and alerts for a particular Storage Grid. Limiting views to specific

VNX arrays can be very helpful when troubleshooting a potential storage issue against a given

Storage Grid composed of multiple arrays.


Figure 4: Unisphere Remote Dashboard

Alert automation and viewing in Unisphere Remote

The Alert dashboard eases monitoring for alerts by array or storage array. With a couple of

mouse clicks, all system alerts can be identified on a single or multiple arrays based on severity

by utilizing tags. These tags enable VNX arrays to be grouped to get a single alert view,

providing an efficent method to review for faults against a particular Storage Grid composed of

multiple arrays. Unisphere Remote enable quick identification of any hardware issue that may

be effecting application performance or availability, thus narrowing the focus of troubleshooting

effort to only the component having problems.


Figure 5: Unisphere Remote Alert Dashboard

VNX Monitoring and Reporting

VNX Monitoring and Reporting (M&R), another tool included with the purchase of a VNX MCx

array, provides performance and capacity reporting capabilities for both block and file storage

types on a VNX array. Each VNX M&R instance can support up to 5 VNX arrays. Note: the full

Watch4Net package on which VNX M&R is based has higher support limits and increased

capabilities, albeit at additional cost. VNX M&R has dozens of default reports that can provide

valuable insight to the performance and capacity utilization of VNX arrays. Examples of the

default performance reports include Storage Pool IOPs bottleneck, LUN performance

Bottlenecks, Disk Utilization Heat Map, and many more. In addition to generating reports on

performance and capacity utilization, inventory reports can also be generated that include all

arrays, firmware levels, and associated storage layout resources (LUNs, RAID Groups, Storage

Pools, Host, Ports, and applications). Along with reporting, VNX M&R can also provide

performance and threshold alerts to both SNMP and SMTP targets. Simple monitoring around

key performance indicators such as response time threshold can provide early warning of

potential issues that will impact Oracle RAC performance.


Figure 6: VNX Monitoring and Reporting Dashboard

EMC Secure Remote Support Gateway

A third tool included with the purchase of any VNX MCx array, EMC Secure Remote Support

Gateway (ESRS) provides an asynchronous messaging system to enable secure encrypted file

transfer for device monitoring of EMC hardware products as well as outbound notification to

EMC of hardware and software faults that require support. The ESRS platform also provides the

remote access needed to support diagnostic activities of EMC support staff to speed time to

resolution of hardware/software related faults that occur. Such automation helps minimize the

time to resolution of any hardware/software problem that may occur on EMC storage products.

ESRS is a critical piece of efficient management of a scale out Storage Grid solution that could

be composed of dozens or even hundreds of storage arrays for the most complex customer

environments.


Figure 7: EMC Secure Remote Support Gateway Archiecture

Conclusion

The VNX MCx platform provides the ideal storage solution to build a scale out, high

performance Oracle RAC data warehouse application. Providing the high bandwidth and

storage capacity demanded by Oracle RAC data warehouses can be very challenging. The

exceptional performance scalability provided by VNX Multi-core I/O technology enables the

solution to scale linearly based on the available disk I/O processing power. This article provides

a starting point to develop a custom building block-based solution that meets all the application

requirements and does so with an easy to manage, high availability solution.


References

EMC documentation utilized for ESRS, VNX M&R, and Unisphere Remote. Figure 4, 5, 6, and 7

screen shots taken from the following documents.

Operation Guide - EMC Secure Remote Support Gateway for Linux Release 2.22 Rev 03

White Paper – EMC Unisphere Remote: Next Generation Storage Monitoring, A Detailed

Review November 2012

EMC VNX Monitoring and Reporting Version 1.1 User Guide P/N 300-014-420

EMC believes the information in this publication is accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION

MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO

THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an

applicable software license.

DESIGN AND ARCHITECTURE OF ORACLE RAC/ASM CLUSTERED …€¦ · approach for a multi-cluster...

Documents

Transcript of DESIGN AND ARCHITECTURE OF ORACLE RAC/ASM CLUSTERED …€¦ · approach for a multi-cluster...