Open Ha Cluster

Discovering Open High Availability Cluster

Robert Sohigian

Sun Microsystems

Outline

What is Open High Availability Cluster?

Why is High Availability Important?

How Solaris Cluster Provides High Availability

Cluster Agents

Cluster in Action

Open-Source Roadmap

How to Use Open HA Cluster

How to Participate

What is Open High Availability Cluster?

Open High Availability Cluster

The open-source code base for SolarisTM Cluster

Based on Solaris Cluster 3.2

Hosted by the HA Clusters community group on opensolaris.org
http://opensolaris.org/os/community/ha-clusters/ohac/

Code available under the CDDL

Why is Sun Open-Sourcing Solaris Cluster?

Share

Public should have access to the source-base

We hope developers will contribute back their agents and other modifications

Open Development

Increase confidence in the cluster code-base

Enhance the code-base with third-party contributions

Why is Sun Open-Sourcing Solaris Cluster?

Shorten Feedback Loop

Easier to incorporate feedback

Find and fix bugs quicker

Early adoption of new features

Because Solaris Cluster is FUN!

Why Is High Availability Important?

Why is HA Important?

Downtime is costly

Failures are inevitable

Hardware, Software, Human Error, Disaster

With a single physical system, single points of failure are catastrophic

Network card dies, CPU misbehaves, disk drive crashes ...

HA Clusters automate the recovery process from inevitable failures, minimizing downtime and cost

Downtime is costy:

Example: Credit card database. If the DB is down, your card will be rejected. If your card is rejected, you'll use a different card instead of waiting for the DB to come back up. The bank loses your business.

Example: Ad-revenue-based web site. If your web server is down, you lose traffic and fail to generate ad revenue for that period of time.

Example: Online vendor. If your web server, business logic, or DB is down, your potential customers might go to a different site. You lose business.

Examples of failure: Hardware failure (CPU, memory, disk, network card, cable, ...), Software bug (application crash, OS panic, ...), Human error (Tripping over power cord, unplugging network cable, ...), Disaster (Earthquake, tornado, hurricane, terrorism, ...)

Why automated recovery: Operator Intervention is Annoying ( No one enjoys being on call 24/7), Operator Intervention is Costly (Overtime pay), Operator Intervention is Slow (Recovery usually slower than when automated)

Hardware Redundancy: A cluster of two or more physical systems can use hardware redundancy to recover from any single point of failure

Can use off-the-shelf hardware

How Solaris Cluster Provides High Availability

Open HA Cluster is the source-code for the Solaris Cluster product. In order to understand Open HA Cluster, we need to cover some background on Solaris Cluster.

Platform for High Availability

Tolerates Single Points of Failure (and some multiple failures)

Hardware redundancy with off-the-shelf hardware

Robust software HA infrastructure

Monitors the health of cluster infrastructure (hosts, shared storage, and network)

Orchestrates recovery of applications and cluster infrastructure (shared storage and IP networking)

Integrated tightly with the Solaris Operating System

Robust membership including quorum to prevent split-brain

Disk Fencing ensures Data Integrity in spite of failures

Solaris Cluster is a General purpose HA Platform for making Solaris applications Highly Available. It's a tightly-coupled distributed system of one or more physical computers. Supports both traditional failover and scalable HA

Product History

SC 3.2 FCS - Dec. 2006

SC 3.0 FCS - Dec. 2000, and several updates/releases in between

Prior products: HA 1.x 1990s, SC 2.y -1990s.

Hardware

Hardware Redundancy: Hosts, Dual private interconnects between hosts (Availability and bandwidth), Public (external to cluster) network: uses Solaris IPMP, Redundant NICs optional, IPMP can monitor single NIC, Storage connected to multiple hosts for availability, Volume manager for disk mirroring

File Systems: Cluster File System (PxFS), QFS for Oracle RAC configurations, ZFS, UFS, VXFS

Tightly integrated with Solaris

Allows the infrastructure to be resilient to load

Exploits kernel hooks for faster failure detection and recovery

Leads to higher and more predictable availability

Solaris Cluster Stack

OperatingSystemCluster InfrastructureAgentsApplications

Heartbeats

Membership

Spend a while on this slide explaining what a cluster is.

There are several (at least two) physical nodes, tightly coupled with software heartbeats.

Solaris Cluster is a layered product on top of Solaris.

Cluster Infrastructure (core) provides High Availability Platform for Applications

Applications run on cluster unmodified (off-the-shelf)

Cluster Agents (also called Data Services) are the glue layer between applications and cluster infrastructure

Custom agent for each application

Agent interacts with cluster core through APIs

Application cannot break into the high availability business on Solaris Cluster without an agent!

HA Failover Service

Application failover

Within nodes or container

Containers failover

Failover IP address

Failover Service

Service is hosted by a primary node in the cluster, with backup capability on one or more secondary nodes.

Exactly one service instance active at a time

Example: Database

Scalable ServiceSoftware Load Balancing

Global Network Service Provides Global IP address with failure protection

Scalable Service

Service is hosted by several nodes in the cluster at the same time, with backup capability on zero or more nodes.

Note that this diagram doesn't show all nodes directly connected to all other nodes, which is required for correct configuration.

Apache (Scalable)


MySQL (Failover)

Example: Apache and MySQL

Service is hosted by several nodes in the cluster at the same time, with backup capability on zero or more nodes.

Heartbeats

Membership

Quorum

Disk Fencing

Resource Group Manager

Resource (application) dependencies

Inter RG dependencies

RG affinities

HA Failover Service

Application failover

Within nodes or container

Containers failover

Failover IP address

Global File ServiceFailover File service

Scalable ServiceSoftware Load Balancing


MonitoringSolaris Cluster Architecture

Talk as little or much as folks are interested on this slide. Some more details:

Cluster Communications: Distributed Object Framework based on CORBA, Called the ORB, Available in kernel and userland; RPC and Doors

Cluster Algorithms: Heartbeats (Monitor nodes in the cluster over the private network, triggering reconfiguration when nodes join/leave, Resilient to load), Membership (Establishes clusterwide consistent cluster membership, Coordinates reconfiguration of other layers ), Quorum (Prevents partitions (split brain, amnesia) in the cluster, Protects against data corruption, Split brain is partition in space, such that cluster partitions can not talk to each other, but are up at the same time, Amnesia is partition in time, such that cluster partitions in different points in time are up, but the later one is not aware of the former and any changes committed to the cluster data by the former., Uses a majority voting scheme, 2 node clusters require a quorum device (an external tie-breaker), Disk Fencing (Used to preserve data integrity, Non cluster nodes are fenced off from updating any shared data)

Membership changes trigger algorithms of upper layers, including the ORB, CCR, Global File System or PxFS, Global Device Service, Global Networking, Resource Group Manager (in user space)

Resource Group Manager (RGM): Rich and extensible framework for plugging applications into Sun Cluster, RGM uses abstraction of a resource to represent an application, RGM interacts with the application Agent, which provides methods for controlling the application (Start, Stop, Monitor, Validate), Closely related resources placed in Resource Groups (Ex. HA-NFS: RG has 3 resources: NFS, IP, Storage), An RG is a basic failover unit, Supports both failover and scalable RGs , Support for rich dependencies between Resources and Rgs, Additional semantics for inter RG dependencies

Cluster Agents

Solaris Cluster Stack

OperatingSystemCluster InfrastructureAgentsApplications

Cluster Agents (Data Services)

Applications run on the cluster unmodified

Cluster agents are the glue layer between applications and cluster infrastructure

Custom agent for each application

Interacts with the cluster core through APIs

Provides start, stop, and other commands specific to the application to be called by the cluster framework

Provides monitor daemon specific to the application

Applications cannot break into the high availability business on Solaris Cluster without an agent!

Intentionally not getting into GDS or agent builder on this slide. That will come in the next slide. The point is to introduce the concept of an agent.Agents also called Data Services.

Agents Development

Several choices available

Sun Cluster Agent Builder

Generic Data Service (GDS)

Data Service Development Library (DSDL)

RGM Application Programming Interface (API)

In the context of making the application service highly available, we have made it easier by delivering predefined application service agents

These are in-house agents and agents developed by ISV themselves

Agents Available Include

Business Logic Tier

HA Sun Java System App Server PE/SE

HA Sun Java System App Server EE

HA Sun Java System Directory Server*

HA Agfa IMPAX

HA BEA Weblogic Server

Scalable Broadvision One-To-One*

HA IBM Websphere MQ

HA IBM Websphere MQ Integrator

IBM Lotus Notes+

HA Oracle Application Server

HA SAP liveCache

HA SAP J2EE Engine

HA SAP Enqueue Server

Scalable SAP

HA Siebel

HA SWIFTAlliance Access

HA SWIFTAlliance Gateway

HA Sybase Adaptative Server*

Web Tier / Presentation

HA Sun Java System Web Server

HA Sun Java System Messaging Server*

HA Sun Java System Message Queue

HA Sun Java System Calendar Server*

HA Sun Java System Instant Messaging
Server*

Scalable Sun Java System Web Server

HA Apache Web/Proxy Server

HA Apache Tomcat

Scalable Apache Web/Proxy Server

Database Tier

Oracle Parallel Server*

HA Oracle9i and Oracle 9i RAC*

HA Oracle 10g and Oracle 10gRAC*

HA Oracle E-business Suite

HA Oracle*

HA Sybase*

HA MySQL

HA SAP/MaxDB Database

HA PostgreSQL

* Not yet open-source

Management Infrastructure Tier

HA Sun N1 Grid Engine

HA Sun N1 Service Provisioning System

HA DNS, HA NFS

HA DHCP

IBM Tivoli+

Mainframe Rehosting (MTP)*

HA Samba

HA Solstice Backup*

HA Solaris Container

HA Symantec NetBackup*

Online Agent list:http://www.opensolaris.org/os/community/ha-clusters/ohac/Documentation/Agents/open-agents/

In the context of making the application service highly available, we have made it easier by delivering predefined application service agents

These are in-house agents and agents developed by ISV themselves

Not all these agents are open-source yet!!!

Open-Source Roadmap

Multiple Source Trees

Solaris Cluster consists of four different source trees

Agents

Core

Geographic Edition

SCATE

Since agents are open-source we'll focus on the agents code details (obviously we can't talk about the other code yet).

What's Open-Source Now

Most of the cluster agents

Geographic Edition

Cluster agents and geo edition G11N

CTI for TET test framework

Based on SCATE (Sun Cluster Automated Test Environment)

Used for both ON and Cluster tests

Test suites for agents and geo edition

More on SCATE:

A suite of automated tests and tools

Distributed test development framework

Distributed test execution framework

Client/server architecture

Easy to plug-in new test suites

Framework now available as CTI for TET (Common Test Infrastructure for Test Environment Toolkit)

Used for both ON and cluster test suites

SCATE Test Assets (not open-source yet)

50+ automated test suites, each with 100s of tests

500+ fault points in the product

350000+ functional tests, 45000+ faults injected

Much More to Come

2008 release source for

Sun Cluster core

Test suites for core

Asynchronously

Docs

Over three million lines of code to be open-sourced!

How to Use Open HA Cluster

In order to understand Open HA Cluster, we need to cover some background on the Solaris Cluster product.

Solaris Cluster Express

A complete version of Solaris Cluster software that runs on Solaris Express

Solaris Cluster Express 02/08 (current release)

Runs on Solaris Express Developer Edition 1/08

Runs on SPARC and 32 or 64-bit x86 hardware

Support available only through community forums

Where to get it
http://opensolaris.org/os/community/ha-clusters/ohac/downloads/

Solaris Cluster 3.2

Most recent release of the Solaris Cluster product

Runs on Solaris 9 and 10

Supported on SPARC and 64-bit x86 hardware

Try it out for free

Where to get it
http://sun.com/software/solaris/cluster/get.html

Build and Run Agents and Geographic Edition

Install

Solaris Express Developer or Community Edition

Compiler

Sun Studio 11, Sun Studio 12, or gcc

Build tools

ON build tools (SUNWonbld)

OHAC build tools (SUNWscbld)

Download source tarballs and archives

One-line nbuild command

Install packages on Solaris Cluster Express or Solaris Cluster 3.2

Build, Install, and Run OHACTS

Download CTI for TET source from testing community
http://opensolaris.org/os/community/testing/testsuites/ctifortet/

Follow CTI for TET instructions to build it

Download OHACTS source
http://opensolaris.org/os/community/ha-clusters/ohac/downloads/

Build and install OHACTS package http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/Tests/buildntest/

Run OHACTS
http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/OHACTS.pdf

How to Participate

Where to Start

HA Clusters Community Group Web Page http://www.opensolaris.org/os/community/ha-clusters/

Open HA Cluster page http://opensolaris.org/os/community/ha-clusters/ohac/

[email protected] mailing listhttp://mail.opensolaris.org/mailman/listinfo/ha-clusters-discusshttp://opensolaris.org/jive/forum.jspa?forumID=195

How to Contribute to Open HA Cluster http://opensolaris.org/os/community/ha-clusters/ohac/contribute/

What You Can Do

Browse the code with OpenGrok

http://src.opensolaris.org/source/xref/ohac/

Join the discussions on ha-clusters-discuss.

File bugs

http://www.opensolaris.org/bug/report.jspa

Fix a bug

http://opensolaris.org/os/bug_reports/oss_bite_size/

Contribute to an existing project

http://www.opensolaris.org/os/projects/

Start a new project

Two new community projects created

HA-Informix

Cluster Agent for Informix Dynamic Server

Created 1st August 2007

Design Doc, Cheatsheet etc. available

http://opensolaris.org/os/project/ha-informix/

HA-xVM

Cluster Agent for Solaris xVM guest domains

http://opensolaris.org/os/project/ha-xvm/

Possible Projects

Develop Agents for

Adabas, BMC Control-M, BMC Patrol Server, BMC Performance Manager, CA Unicenter, Citrix, Connect Direct, EMC Documentum, FileNet, IBM Tivoli Client, IBM Tivoli Server, IBM Tivoli Workload Scheduler, IBM Websphere Application Server, Informatica, Java CAPS, Java DB, JBOSS, Netbackup, Netbackup Client, Open LDAP, Sleepy cat DB, Sun Java System Identity Server, Sun Java System Portal Server, Sybase Replicator agent, TSM Backup client, UC4

More project suggestions for Open HA Cluster http://opensolaris.org/os/community/ha-clusters/ohac/contribute/agents-suggested-projects/

Or propose your own project!

Thank you!

Robert [email protected]

Based on Slides by
[email protected]

open artwork and icons by chandan:
http://blogs.sun.com/chandan

Click to edit the title text format

Click to edit the outline text format

Second Outline Level

Third Outline Level

Fourth Outline Level

Fifth Outline Level

Sixth Outline Level

Seventh Outline Level

Eighth Outline Level

Ninth Outline Level

USE

IMPROVE

EVANGELIZE

Click to edit the notes format

Click to edit the title text format



Third Outline Level


Fifth Outline Level

Sixth Outline Level



Ninth Outline Level

USE

IMPROVE

EVANGELIZE




Third Outline Level


Fifth Outline Level

Sixth Outline Level



Ninth Outline Level

USE

IMPROVE

EVANGELIZE


Open Ha Cluster

Documents

Transcript of Open Ha Cluster