Creating a Maximum Availability Architecture with SPARC SuperCluster


Creating a High Availability Architecture with SPARC SuperClusterCarter WrightDatabase Engineer, Computer Knowledge, Inc.

Bruce ChapmanPrincipal Software Engineer, Oracle

Dean HalbeisenSolutions Manager, Oracle


ProgramAgenda

SPARC SuperCluster High Availability

Implementation


Optimized Solutions High Availability

Examples

Next Steps / Resources



Leverage the HA features built in to the SPARC SuperCluster– Oracle Real Application Cluster(RAC), Oracle Exadata Storage Server,

Oracle Automatic Storage Management(ASM), Network and Power redundancies

– Clustering of applications in Oracle VM Server For SPARC and Oracle Solaris Zones on SPARC T4-4 nodes

Use of Oracle Data Guard Use of Oracle RMAN Use of Sun ZFS Storage Appliance 7420 and ZFS Replication

Large Government Law Enforcement Agency Implementation of HA



12,000 to 15,000 concurrent users worldwide during peak hours

Database greater than 1.6TB and growing

1500 to 2000 SQL executions per second per RAC node during peak hours (~4000 SQL execs per second combined)

Greater than 1.5 million blocks per second read from each SGA during peak hours ( > 3 million blocks per second combined )

900K to 1.2MB of redo written per second (combined RAC nodes)

Query response time typically less than 300ms

Data Guard latency less than three seconds to standby site

ZFS replication latency less than three seconds to standby site

Each RAC node CPU utilization at peak load 15% to 20%

Workload Overview


SPARC SuperCluster High AvailabilitySPARC SuperCluster Configuration

ExternalServer1

ExternalServer3

ExternalServer2

10GbE ZFS 7420 StorageRMAN BackupsData Pump Dir

InfiniBandN

od

es 1

& 2

No

des

3 &

4

Infi

niB

and

Solaris 10

600G10K SAS

App SvrApp Zone

App Zone

Exadata Data Cells216TB RawASM High

72TB Usable3 X ASM Disk Groups

Solaris 10

App Zone

Solaris 11



SuperCluster installation does provide a fully functional Oracle RAC install, but it has to be configured to meet your applications needs

– Database Hardening must be performed– Listener configurations (scan and local) can be modified– You configure your RAC Workload Services– Network settings – 10Gb and 1Gb interfaces only

TCP window, MTU’s Leave IB settings alone

– Backup and recovery is your responsibility– Application tuning is your responsibility

Database Configuration



– OLTP or Data Warehouse (or both)– SGA Settings can be modified

We changed only a few settings

– SGA, PGA Aggregate Target– Data Guard parameters

SPARC SuperCluster uses Auto SGA and Auto PGA, not full AMM (see pg 18-18 11g Concepts Guide)

Can change the quantity and size of Redo Logs (if needed) Users, tablespaces, datafiles

– Roles and privileges– Datafile size and quantity

Database Configuration


SPARC SuperCluster High AvailabilityWorkload Services

Smaller Apps

Data Guard

Non-RAC COTS

Data Guard

Single Node Services

Main App 1

RMAN Service

Main App 2

Load Balanced Services

SCAN Listeners

SCAN + Service Name

Intranet

12K-15K Active Concurrent Users Accessed World Wide



Data Guard setup– Physical Standby, High Performance Mode, Async

Multi-instance Primary, single instance standby Active Guard enabled at standby site Use standby for

– Reporting and Read-only access– Smoke testing application deployments

Make standby a temporary Primary and restore back to standby using Flashback Database

Use standby as the Last deployment before deployment to prod site– Other live applications to leverage investment

Data Guard on SPARC SuperCluster


SPARC SuperCluster High AvailabilityReplication Strategy

Primary SPARC SuperCluster

Standby SPARCSuperCluster

ZFS Storage Appliance 7420

Data Guard ReplicationDatabase Content

ZFS ReplicationExternal Content

ZFS Storage Appliance 7420

Latency to Remote Site <3 Seconds

Latency to Remote Site <3 Seconds



Backupsets stored on ZFS 7420 (not in ASM)– Accessible during crisis

– Leave space to database in Exadata

– Backup file system mounted to both nodes for parallelism

– ZFS RAID 10

Use RAC Workload Service– Allows RMAN parallelism across RAC

rman target sys/<pw>@rman_svc

Improves speed

Parallelism 16 with 16 Channels allocated– RMAN will allocate 8 on node 1, 8 on node2

RMAN Configuration

ZFS 7420 StorageRMAN Backups

No

de

2N

od

e 1

Solaris 10

App Zone

Solaris 11

Solaris 10

App Zone

Solaris 11

InfiniBand

RM

AN

Ser

vice

s



Level 0 weekly– 7 day recovery window

Level 1 nightly– Cumulative Incremental

Backup both primary and standby sites separately

RMAN Backup Operations



Recovery Catalog is required Flashback database enabled at both sites db_unique_name required Backups taken at both sites

– BACKUP and RESTORE command uses TAG ‘system_name’ option when creating backups

Distinguishes each site in Recovery Catalog

Archivelog deletion policy on Primary– Ensures archivelogs are not deleted by RMAN before getting to standby site

CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;

RMAN Configuration and Management in a RAC Environment



Grid Control 12c– Good graphs and drill down capabilities

Custom Scripts (bash)– Exadata Cell monitoring –

No Grid 12c plug-in yet for SPARC SuperCluster with Exadata– Alert monitoring to send notifications to our NOC

Monitoring and Management



Early adopter of SPARC SuperCluster and it was not your common install being a customer engineered application, had to train a new team with clearance to work on the systems to perform the install

Purchased three SPARC SuperCluster systems in December 2011 Pre-Prod, Production, Standby DR site

First system delivered early February, next system two weeks later, third system two weeks after that

Each system was installed and running our application within three weeks of each delivery.

Within a 9 week period, we had three fully operational SuperClusters in three different data centers.

Went live production in July of 2012. Successful worldwide launch

Instantiated DR site with Oracle Data Guard shortly thereafter.

Time to Deployment for an Engineered Solution



SPARC SuperCluster High Availability White papers on www.cknowledge.com

– How To Guide: RAC Workload Services – 11gR2– How To Guide: Building Data Guard– RMAN Quick Reference

Knowledge to Share

http://www.cknowledge.com/


ProgramAgenda


Implementation



Examples



SPARC SuperCluster: High AvailabilityHardware HA Features

Dual Power Distribution Units Both connected to each server or switch in the SuperCluster, each with redundant power supplies.

Ten Gigabit Ethernet NetworkingEach logical 10GbE interface has two physical ports for connection to separate external lab switches.

Infiniband Networking- 3 IB switches for availability / growth.

- Each logical IB connection to T4-4, Exadata DB Storage Cells, and 7320 Storage Appliance has two physical InfiniBand ports.

Mirrored Disks All OS Disks On T4-4 Domains are mirrored.

SPARC T4-4 HA FeaturesHot swap redundant PS, hot swap PCI-E, DIMM off lining, predictive self healing

7320 ZFS Storage Appliance-Two Storage head nodes each with redundant networking, SCSI.-Configurable RAID / spare options for shares. Database Storage CellsEach with redundant IB and configurable disk mirroring.


SPARC SuperCluster: High AvailabilitySoftware HA

Multiple RAC instancesEnabling DB to tolerate and react to instance and node failures automatically with zero to near zero impact on applications.

Oracle Grid Infrastructure Clusterware and ASM provide abstraction of DB storage cell nodes and disks, also able to continue functioning despite disk or DB storage node failure.

Oracle Solaris Cluster One of several high availability options for application domains. Configurable number of Domains for horizontal scaling and failover.


ProgramAgenda


Implementation



Examples



T4-4 Node 4

Oracle Solaris

Oracle Solaris 11

Database Domain

GP Domain

T4-4 Node 3

Oracle Solaris

Oracle Solaris 11

Database Domain

GP Domain

InfiniBand Network

Oracle Exadata Storage Servers

10 GbE Data Center Network

T4-4 Node 2

Oracle Solaris

T4-4 Node 1

Oracle Solaris 11

Database Domain

GP Domain

OracleSolarisCluster

Oracle RAC

11g R2

High Availability

Scale

High Availability on SPARC SuperClusterOracle Optimized Solutions

Oracle Solaris

Oracle Solaris 11

Database Domain

GP Domain

T4-4 Node 2


High Availability / Fail Over Test ResultsExtremely fast failure detection and service recovery times

Detection

SECONDS

Web server node failure 110

61

OPMN/OC4J failure

General purpose domain failure

Database domain failure

0

Recovery

120

5 10


Driving Risk Down • Driving Innovation Up • Active Partnership with Oracle

Down • Driving Innovation Up • Active Partnership with Oracle • Driving Risk

Up • Active Partnership with Oracle • Driving Risk Down • Driving Innovation











Tested, Secure, Flexible

Driving Risk Down

Driving Innovation Up

Active Partnership with Oracle


ProgramAgenda


Implementation



Examples



SPARC SuperCluster Conference SessionsSession Date Time Room

Accelerate Oracle E-Business Suite Deployment with SPARC SuperCluster Monday 12:15PM Moscone West - 2001

SPARC SuperCluster: Architecture, Roadmap, and Partner Services Opportunities Monday 12:15PM Marriott Marquis – Golden Gate B

My Oracle Support: Tips and Techniques for Getting the Best Hardware Support Monday 1:45PM Moscone West - 2001

Top 10 Database Performance Tips for SPARC Systems Running Oracle Solaris Monday 1:45PM Moscone West - 2000

Building a Flexible Enterprise Cloud Infrastructure on Oracle SPARC Systems Monday 4:45PM Moscone West - 2001

Database and Application Consolidation on SPARC SuperCluster Monday 4:45PM Moscone West - 2000

Oracle's SPARC Server Strategy Update Tuesday 10:15AM Moscone West - 2001

Oracle Engineered Systems Backup and Recovery Best Practices Tuesday 10:15AM Moscone South - 252

Transform Data Center TCO with Oracle Optimized Servers: A Customer Panel Tuesday 11:45AM Moscone West - 2001

Deep Dive into SPARC SuperCluster: Engineered for Extreme Efficiency Tuesday 1:15PM Moscone West - 2001

Accelerate Your SAP Landscape with the Oracle SPARC SuperCluster Tuesday 5:00PM Moscone West - 2001

High Performance Security for Oracle Applications Using SPARC T4 Systems Tuesday 5:00PM Moscone West - 2000

SPARC Processor Update Wednesday 10:15AM Moscone West - 2000

Creating a Maximum Availability Architecture with SPARC SuperCluster Wednesday 11:45AM Moscone West - 2000

Improve PeopleSoft HCM Performance and Reliability with SPARC SuperCluster Wednesday 5:00PM Moscone South - 200

Managing Sun Servers and Oracle Engineered Systems with Oracle Enterprise Manager Thursday 12:45PM Moscone West - 2000


SPARC SuperCluster DemosDemo Location

Accelerating SAP Queries and Reporting on SPARC SuperCluster Moscone South -102

Bullet Proof E-Business Suite on SPARC SuperCluster Moscone South -101

Bullet proof PeopleSoft on SPARC SuperCluster Moscone South-131

Consolidate your Datacenter: Enterprise Cloud Infrastructure on SPARC Moscone South-150

Explore the Benefits of Oracle SPARC SuperCluster Moscone South-104

Mission Critical Availability and Disaster Recover for Your Cloud Moscone South-116

Mission Critical SPARC-Powered Infrastructure as a Service Moscone South-157

Oracle SPARC SuperCluster: World’s First General Purpose Engineered System Moscone South-106

Oracle SPARC SuperCluster T4-4 to Improve Efficiency and Cut Costs Moscone South-105

The Data Center Superhero: Backup and Recovery Systems Moscone South-107

Transform an Enterprise Cloud Infrastructure: Oracle SPARC SuperCluster Moscone South-130

Transform and Optimize Revenue Management with SPARC SuperCluster Moscone South-108

Engineered System Showcase - SPARC SuperCluster Consolidation Moscone North Lower Lobby


Resources

Computer Knowledge White Papers– http://www.cknowledge.com

Oracle Maximum Availability Architecture Guides– http://www.oracle.com/technetwork/database/features/availability/maa-090

890.html

Optimized Solutions– http://www.oracle.com/us/solutions/oos/overview/index.html

In depth information

http://www.cknowledge.com/

http://www.oracle.com/technetwork/database/features/availability/maa-090890.html




Graphic Section Divider

Creating a Maximum Availability Architecture with SPARC SuperCluster

Technology

Transcript of Creating a Maximum Availability Architecture with SPARC SuperCluster