hk.emc.com · EMC AutoStart Concepts Guide 1 CONTENTS Chapter 1 AutoStart Domains EMC...

EMC® AutoStart™

Version 5.5.0

Concepts GuideREV A02

EMC AutoStart Concepts Guide2

Copyright ©1998- 2013 EMC Corporation. All rights reserved. Published in the USA.

Published December, 2013

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date regulatory document for your product line, go to the technical documentation and advisories section on the EMC online support website.

CONTENTS

Chapter 1 AutoStart Domains

EMC AutoStart............................................................................................... 8 AutoStart domains ........................................................................................ 8

Domain networks .................................................................................... 9Planning a domain's networks ................................................................ 9Multiple domains .................................................................................. 10Domain name format............................................................................. 10Domain tabs in the AutoStart console ................................................... 10What to consider when working with an AutoStart domain .................... 10

Redundant networks in the domain............................................................. 11Failover ................................................................................................. 11Failback ................................................................................................ 11

Heartbeats.................................................................................................. 12 How AutoStart detects a failure ................................................................... 12

Failure detection scenario ..................................................................... 12Networks and heartbeats ...................................................................... 13Examples of how AutoStart detects a failure.......................................... 13

AutoStart domain networks......................................................................... 17WAN configuration ................................................................................ 18Multicast versus point-to-point communication..................................... 18

Isolation detection...................................................................................... 19Here's how it works ............................................................................... 19Isolation addresses............................................................................... 20If you don't set up isolation detection ................................................... 20Isolation scripts .................................................................................... 21

Verification networks for the AutoStart domain............................................ 22 Symmetrix networks for the AutoStart domain............................................. 23 Mirroring the Windows Registry Monitor ...................................................... 24

Why use AutoStart's Registry mirroring? ................................................ 24The processes ....................................................................................... 24How AutoStart's registry mirroring works ............................................... 24Mirroring methods ................................................................................ 25Registry checkpoint............................................................................... 25Set up and use ...................................................................................... 25

User access and security ............................................................................. 26Initial administrator for the AutoStart domain........................................ 26Levels of security .................................................................................. 26Administrator level................................................................................ 26Operator level ....................................................................................... 27User level .............................................................................................. 27Access from locations ........................................................................... 27Additional, underlying OS-level rights ................................................... 27

Using definition files with AutoStart ............................................................ 28

Chapter 2 Nodes and Their Agents, NICs, and IPs

Nodes......................................................................................................... 30Node tabs in the AutoStart console ....................................................... 30

AutoStart agents ......................................................................................... 30Primary agents ...................................................................................... 30

EMC AutoStart Concepts Guide 1

Contents

Secondary agents ................................................................................. 31 NICs............................................................................................................ 31

AutoStart detects NICs .......................................................................... 32NIC group .............................................................................................. 32Unplumbed NICs ................................................................................... 32Checklist for setting up NICs.................................................................. 32Viewing NICs in the AutoStart console ................................................... 33NIC tabs in the AutoStart console .......................................................... 33

NIC testing .................................................................................................. 33Testing on UNIX..................................................................................... 34Testing on Windows .............................................................................. 34Custom NIC Testing ............................................................................... 34

When a NIC fails.......................................................................................... 35NIC-to-NIC failover................................................................................. 36

Usable, standby, and reserved NICs ............................................................ 38 NIC groups .................................................................................................. 38

Auto-creation of a NIC group.................................................................. 39Configuring a NIC group......................................................................... 39Unplumbed NICs ................................................................................... 39NIC group tabs in the AutoStart console ................................................ 40

IP addresses ............................................................................................... 40The details ............................................................................................ 40Managed IP addresses for managed applications.................................. 40If there is a NIC failure ........................................................................... 41Use the IP address or name service entry .............................................. 41Base IP versus managed IP.................................................................... 42MAC addresses on UNIX nodes.............................................................. 42Setting the IP limit when you install an agent on a Windows node ......... 42If you remove an IP address................................................................... 42IP address tabs in the AutoStart console ............................................... 43Example of IP address operation ........................................................... 43

Node aliases for Windows ........................................................................... 43Node alias tabs in the AutoStart console ............................................... 46

Node proxies............................................................................................... 46Node proxy tabs in the AutoStart console .............................................. 46

Chapter 3 State Monitors

State monitors ............................................................................................ 48User-defined versus built-in monitors.................................................... 48A more detailed look at state monitors .................................................. 49Writing monitor tests............................................................................. 49State monitor tabs in the AutoStart console .......................................... 50

Response monitor tests .............................................................................. 50When a response monitor reports a failure ............................................ 51Built-in versus custom response monitors............................................. 51Guidelines for custom response monitors ............................................. 51

Existence monitor tests ............................................................................... 52Built-in versus custom existence monitors ............................................ 52The built-in existence monitor ............................................................... 52Custom existence monitors for processes and process proxies.............. 53Guidelines for custom existence monitors ............................................. 53Existence monitor timeout..................................................................... 54

2 EMC AutoStart Concepts Guide

Contents

Chapter 4 Data Sources

Data sources............................................................................................... 56Data access .......................................................................................... 56Attaching .............................................................................................. 56List of data sources ............................................................................... 56

The anatomy of an attach ............................................................................ 57The response and existence tests.......................................................... 57Windows Network Share, UNIX file system, or Shared Disk Device for Windows data source types................................................................... 58

Parallel attach of data sources .................................................................... 58Used with a data source checkpoint ...................................................... 59Example................................................................................................ 59

AutoStart data sources for RepliStor 6.x ...................................................... 61Working with RepliStor 6.x data sources in AutoStart ............................ 61RepliStor server failure .......................................................................... 62AutoStart data source for RepliStor 6.x data source tabs in the AutoStart console ................................................................................................. 62

AutoStart data sources for RepliStor 5.x ...................................................... 62AutoStart Data Source for RepliStor 5.x Data Source tabs in the AutoStart console ................................................................................................. 63

Composite data sources.............................................................................. 63Failover ................................................................................................. 63Critical Composite data sources ............................................................ 63Environment variables for Composite data sources ............................... 64Composite data source tabs in the AutoStart console............................ 64Example of failover using a Composite data source ............................... 64Recovery from a remote failover node.................................................... 65

EMC Mirroring for Windows data source ...................................................... 65EMC Mirroring for Windows ................................................................... 66Initial mirroring activity ......................................................................... 66Subsequent changes ............................................................................ 67EMC Mirroring for Windows data source tabs in the AutoStart console... 67

EMC MirrorView mirroring data source......................................................... 67MirrorView/S......................................................................................... 67MirrorView/A......................................................................................... 68You must complete the checklist........................................................... 68Basic MirrorView information ................................................................ 68EMC MirrorView data source tabs in the AutoStart console .................... 69MirrorView data source features and terminology.................................. 69

EMC SRDF Mirroring data source ................................................................. 70SRDF Mirroring Data Source tabs in the AutoStart console ..................... 71SRDF/S versus SRDF/A .......................................................................... 71A typical SRDF configuration.................................................................. 72Example................................................................................................ 73SRDF/Synchronous ............................................................................... 73SRDF/Asynchronous.............................................................................. 75Personality swap illustrated .................................................................. 77Device groups ....................................................................................... 78

Mount points data source for Windows ....................................................... 79Before you begin ................................................................................... 79Tabs in the AutoStart console................................................................ 79

Shared Disk Device for Windows data sources............................................. 80Shared Disk Device for Windows data source tabs in the AutoStart console 81

VERITAS Volume Manager for Windows data sources ................................... 81


Contents

VERITAS Volume Manager for Windows data source tabs in the AutoStart console ................................................................................................. 81

VERITAS Volume Manager for Solaris data sources....................................... 81Using in a resource group...................................................................... 82VERITAS Volume Manager for Solaris data source tabs in the AutoStart console ................................................................................................. 82

Windows Network Share data sources......................................................... 82Windows Network Share data source tabs in the AutoStart console....... 83

AIX Volume Group data sources................................................................... 83Requirements for an AIX Volume Group data source .............................. 83AIX Volume Group data source tabs in the AutoStart console ................ 84

HP-UX LVM data sources.............................................................................. 84HP-UX LVM data source tabs in the AutoStart console ........................... 85

NFS data sources ........................................................................................ 85NFS data source tabs in the AutoStart console ...................................... 85

Sun Solstice DiskSuite data source ............................................................. 85Sun Solstice DiskSuite data source tabs in the AutoStart console ......... 86

UNIX file system type data sources.............................................................. 86Tabs for UNIX file system type data sources in the AutoStart console..... 86

Chapter 5 Resource Groups

Resource groups ......................................................................................... 88Resource groups for Microsoft IIS sites.................................................. 88Modules................................................................................................ 89Resource group tabs in the AutoStart console ....................................... 89

The preferred nodes list .............................................................................. 89Nodeless resource groups..................................................................... 89Sequence is important .......................................................................... 90Nodes are available to all resource groups ............................................ 90Balance the load ................................................................................... 90Create an alternate list of nodes............................................................ 90Use a rule to determine the node........................................................... 91Setting up a preferred nodes list ........................................................... 91Limits when using AutoStart modules ................................................... 91Node groups ......................................................................................... 92

Resources in a resource group..................................................................... 93No manually controlled resources allowed ............................................ 94Resources in multiple resource groups.................................................. 94Startup and shutdown sequences ......................................................... 94

Chapter 6 Modules

AutoStart modules ...................................................................................... 98Granting user access if you use modules............................................... 98EMC does not support customized module instances............................ 99

SQL Server 2005 module ............................................................................ 99Failure detection ................................................................................... 99Failover ................................................................................................. 99Failback .............................................................................................. 100What to do .......................................................................................... 100Tabs in the AutoStart console.............................................................. 100Failover configurations for the SQL Server 2005 module...................... 101

Microsoft Exchange 2003 module ............................................................. 103The resource group ............................................................................. 103


Contents

Virtual nodes ...................................................................................... 104Tabs in the AutoStart console.............................................................. 104Data sources you can use with the Exchange 2003 module ................. 104Server configurations and data access with the Exchange 2003 module..... 105EMC Mirroring for Windows' use with the Exchange 2003 module ....... 105Shared disk device for Windows' use with the Exchange 2003 module 106EMC SRDF Mirroring's use with the Exchange 2003 module ................. 107

Microsoft IIS module ................................................................................. 109Requirements...................................................................................... 109Before you begin ................................................................................. 109

Windows Print Services module ................................................................ 110Tabs in the AutoStart console.............................................................. 110Failover configurations for Windows Print Services .............................. 110

Oracle module for Windows ...................................................................... 112EMC Mirroring for Windows data source for use with Oracle on Windows.... 112Shared Disk Device for Windows data source for use with the Oracle module ............................................................................................... 113EMC SRDF Mirroring data source for use with the Oracle module on Windows............................................................................................. 114Tabs in the AutoStart console.............................................................. 116

Oracle module for UNIX ............................................................................. 116Tabs in the AutoStart console.............................................................. 116Failover configurations for Oracle modules on UNIX............................. 117

Chapter 7 Processes and Services

Processes, services, and proxies ............................................................... 120Proxy processes: Node proxies and process proxies............................ 120Process definitions.............................................................................. 121Service and process failure ................................................................. 121

Processes ................................................................................................. 121Before you begin ................................................................................. 122Process tabs in the AutoStart console ................................................. 122

Process proxies......................................................................................... 123Process proxy tabs in the AutoStart console ........................................ 123

Configurations for processes and process proxies ..................................... 124What is on the configuration?.............................................................. 124Tabs for a configuration....................................................................... 124Troubleshooting configurations........................................................... 125

Services.................................................................................................... 125Prerequisites for services .................................................................... 125Service tabs in the AutoStart console .................................................. 126

Utility processes ....................................................................................... 126Utility process tabs in the AutoStart console ....................................... 126

Chapter 8 Rules, Triggers, and Programming Tools

Rules ........................................................................................................ 128Why use a rule? ................................................................................... 128Some basics about rules ..................................................................... 128What makes up a rule? ........................................................................ 129Rule tabs in the AutoStart console....................................................... 130How a rule runs ................................................................................... 130


Contents

The rule interpreter.............................................................................. 132 Triggers..................................................................................................... 133

Triggers and rules................................................................................ 133Types of triggers.................................................................................. 134What you should know about using triggers ........................................ 134Trigger tabs in the AutoStart console ................................................... 135Event-log triggers ................................................................................ 135Scheduled triggers .............................................................................. 136Sensor-based triggers ......................................................................... 137On-demand triggers ............................................................................ 141

Sensors .................................................................................................... 141AutoStart-supplied sensors and user-defined sensors......................... 142Sensors provided by proxies on behalf of nodes and processes .......... 142Sensor classes for processes .............................................................. 142

Actuators .................................................................................................. 143 Planning a rule.......................................................................................... 144

How to plan a rule ............................................................................... 144Configuring triggers............................................................................. 145Associating triggers with rules............................................................. 146Identifying details ............................................................................... 146Debugging .......................................................................................... 147Where the rule runs............................................................................. 148When the trigger is delivered............................................................... 148Rule execution guarantees .................................................................. 148Resource and node state updates ....................................................... 149

Guidelines for writing rule text .................................................................. 149Understand the problem and the conditions that cause it ................... 150Sample ............................................................................................... 150Review the AutoStart demo and rule tutorial........................................ 150Get to know Perl .................................................................................. 151Gain a basic understanding of the AutoStart API extensions you'll be using 151Node independence............................................................................ 151Access to AutoStart's database ........................................................... 151Remember: rules are single-threaded.................................................. 152Allow the AutoStart agents to run ........................................................ 152Understand the use for actuators ........................................................ 152Understand the triggers you are using ................................................. 152Some rule basics for using triggers...................................................... 153Generate visible output from a rule ..................................................... 154Sample ............................................................................................... 154

AutoStart's API subroutines ...................................................................... 154 Use APIs to make processes AutoStart-aware ............................................ 158

Compiling and linking an aware application program .......................... 158Use a proxy process to publish sensors and actuators......................... 158When the aware process executes....................................................... 159Aware application class ...................................................................... 160

AutoStart's global variables ...................................................................... 160Referencing a global variable .............................................................. 161Perl scripts versus rules ...................................................................... 161Predefined global variables in AutoStart.............................................. 161Example for combining global variables in a rule................................. 162

Quick guide: Resource groups ................................................................... 179


Title Page

FIGURES

1 Sample AutoStart domain network configuration......................................................... 142 Example of a single network failure ............................................................................. 143 Example of dual network failure .................................................................................. 154 Example of total network failure .................................................................................. 165 Test timeout................................................................................................................ 356 Node alias in the Windows Network Neighborhood ..................................................... 457 Composite data source example ................................................................................. 658 MirrorView/Synchronous example............................................................................... 689 Typical SRDF configuration .......................................................................................... 7210 SRDF example ............................................................................................................. 7311 SRDF/Synchronous ..................................................................................................... 7412 SRDF/Asynchronous.................................................................................................... 7513 SRDF personality swap ................................................................................................ 7714 SRDF device groups..................................................................................................... 7815 Node groups displayed in the AutoStart console ......................................................... 9216 Active/passive configuration for the SQL module ...................................................... 10117 Active/active configuration of the SQL module .......................................................... 10218 Exchange 2003 using EMC Mirroring for Windows ..................................................... 10619 Exchange 2003 using a shared disk device ............................................................... 10720 Exchange 2003 using SRDF Mirroring ........................................................................ 10821 WPS with shared disk configuration .......................................................................... 11122 WPS with cached data stored to a mirrored volume ................................................... 11223 Oracle configuration with EMC Mirroring on Windows................................................ 11324 Oracle configuration with a Windows shared disk...................................................... 11325 Multinode Oracle configuration with SRDF................................................................. 11526 Shared disk Oracle configuration on UNIX ................................................................. 11727 SRDF Oracle configuration on UNIX............................................................................ 11828 Triggers and rules...................................................................................................... 13429 Rules driven by sensor-based triggers ....................................................................... 13830 Planning a rule .......................................................................................................... 14731 The aware process executes...................................................................................... 15932 The AutoStart Console............................................................................................... 166


Figures


PREFACE

As part of an effort to improve its product lines, EMC periodically releases revisions of its software and hardware. Therefore, some functions described in this document might not be supported by all versions of the software or hardware currently in use. The product release notes provide the most up-to-date information on product features.

Contact your EMC representative if a product does not function properly or does not function as described in this document.

Note: This document was accurate at publication time. New versions of this document might be released on the EMC online support website. Check the EMC online support website to ensure that you are using the latest version of this document.

PurposeThis document describes how to configure and use EMC AutoStart.

AudienceThe information in this guide is intended for use by system administrators who are responsible for installing software and maintaining the servers and clients on a network.

Related documentationAlso refer to EMC AutoStart online help for a complete listing of steps for setting up and maintaining your AutoStart installation. You can access online help from the AutoStart console, after you have installed AutoStart on a node.

AutoStart documents that you want to have at hand while working with the AutoStart include the latest versions of the following additional guides:

◆ EMC AutoStart Administrator’s Guide

◆ EMC AutoStart Installation Guide

◆ EMC AutoStart Compatibility Guide

◆ EMC AutoStart Release Notes

◆ EMC AutoStart online help

Note: AutoStart contains a component that enables integration with EMC DiskXtender. Instructions for using this component are provided in the EMC DiskXtender 6.2 Installation Guide.

The latest updates of product documentation are available at:

https://support.EMC.com

Conventions used in this document EMC uses the following conventions for special notices:



Preface

CAUTION, used with the safety alert symbol, indicates a hazardous situation which, if not avoided, could result in minor or moderate injury.

Note: A note presents information that is important, but not hazard-related.

IMPORTANT

An important notice contains information essential to software or hardware operation.

Typographical conventions

EMC uses the following type style conventions in this document:

Normal Used in running (nonprocedural) text for:• Names of interface elements, such as names of windows, dialog boxes,

buttons, fields, and menus• Names of resources, attributes, pools, Boolean expressions, buttons,

DQL statements, keywords, clauses, environment variables, functions, and utilities

• URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, and notifications

Bold Used in running (nonprocedural) text for names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system calls, and man pages

Used in procedures for:• Names of interface elements, such as names of windows, dialog boxes,

buttons, fields, and menus• What the user specifically selects, clicks, presses, or types

Italic Used in all text (including procedures) for:• Full titles of publications referenced in text• Emphasis, for example, a new term• Variables

Courier Used for:• System output, such as an error message or script• URLs, complete paths, filenames, prompts, and syntax when shown

outside of running text

Courier bold Used for specific user input, such as commands

Courier italic Used in procedures for:• Variables on the command line• User input variables

< > Angle brackets enclose parameter or variable values supplied by the user

[ ] Square brackets enclose optional values

| Vertical bar indicates alternate selections — the bar means “or”

{ } Braces enclose content that the user must specify, such as x or y or z

... Ellipses indicate nonessential information omitted from the example


Preface

Revision HistoryThe following table presents the revision history of this document:

Where to get helpEMC support, product, and licensing information can be obtained on EMC Online Support, as described next.

Note: To open a service request through EMC Online Support, you must have a valid support agreement. Contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account.

Product information

For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to EMC Online Support (registration required) at:


Technical support

EMC offers a variety of support options.

Support by Product. EMC offers consolidated, product-specific information on the Web at:

https://support.EMC.com/products

Table 1 Revision History

Revision Description and/or Change Enginuity Operating Environment

A01 First Release 5876

A02 Second Release 5876


https://support.EMC.com/products

http://support.EMC.com

Preface

The Support by Product web pages offer quick links to Documentation, White Papers, Advisories (such as frequently used Knowledgebase articles), and Downloads, as well as more dynamic content, such as presentations, discussion, relevant Customer Support Forum entries, and a link to EMC Live Chat.

EMC Live Chat — Open a Chat or instant message session with an EMC Support Engineer.

eLicensing support

To activate your entitlements and obtain your Symmetrix license files, visit the Service Center on https://support.EMC.com, as directed on your License Authorization Code (LAC) letter emailed to you.

For help with missing or incorrect entitlements after activation (that is, expected functionality remains unavailable because it is not licensed), contact your EMC Account Representative or Authorized Reseller.

For help with any errors applying license files through AutoStart, contact the EMC Customer Support Center.

If you are missing a LAC letter, or require further instructions on activating your licenses through the Online Support site, contact EMC's worldwide Licensing team at [email protected] or call:

◆ North America, Latin America, APJK, Australia, New Zealand: SVC4EMC (800-782-4362) and follow the voice prompts.

◆ EMEA: +353 (0) 21 4879862 and follow the voice prompts.

Your commentsYour suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Send your opinions of this document to:

[email protected]



CHAPTER 1AutoStart Domains

This chapter describes redundant networking strategies for an AutoStart domain, and includes discussions of isolation and verification networks, heartbeats, and network interfaces:

◆ EMC AutoStart......................................................................................................... 14◆ AutoStart domains .................................................................................................. 14◆ Redundant networks in the domain......................................................................... 17◆ Heartbeats.............................................................................................................. 18◆ How AutoStart detects a failure ............................................................................... 18◆ AutoStart domain networks..................................................................................... 23◆ Isolation detection.................................................................................................. 25◆ Verification networks for the AutoStart domain........................................................ 29◆ Symmetrix networks for the AutoStart domain......................................................... 29◆ Mirroring the Windows Registry Monitor .................................................................. 30◆ User access and security ......................................................................................... 32◆ Using definition files with AutoStart ........................................................................ 34

AutoStart Domains 13

AutoStart Domains

EMC AutoStartEMC® AutoStart™ delivers availability and automation of enterprise applications such as ERP, database, email, and web applications in complex, heterogeneous environments through a central management interface. AutoStart:

◆ Monitors the health and performance of applications, and helps avoid failures.

◆ Remedies failures by providing alerts and automated remediation when resources begin to run out, for fastest possible time-to-recovery.

◆ Re-provisions and restarts services when failures occur, automating process control and data management.

Using AutoStart, you can safeguard mission-critical applications and resources from events that can impact availability and ultimately reduce business effectiveness.

For the current list of operating systems and product supported for use with AutoStart, refer to EMC Online Support http://support.EMC.com.

IMPORTANT

Some features described in this guide are specific to certain operating systems.

AutoStart domainsAn EMC AutoStart domain is a group of nodes that have AutoStart agents installed on them; they work together to provide a highly available environment.

Note: An AutoStart domain consists of a group of nodes that AutoStart manages; it is different than a Windows networking domain.

When you install and run an AutoStart agent on a machine, that machine becomes a node. You identify the node name and AutoStart domain name when you install an AutoStart agent onto the machine. Then, when you're ready, you use the AutoStart console to define all of the domain's nodes to the AutoStart database. For more, see “Nodes” on page 36.

An AutoStart domain can have as few as 2 nodes, or can have as many as 5 nodes running primary agents and an additional 5 nodes running secondary agents. A node can belong to one, and only one, AutoStart domain. All of the nodes in the AutoStart domain share characteristics such as an AutoStart database, event log, and failure detection settings. All nodes run an agent, but not all run primary agents; for more information, see “AutoStart agents” on page 36.

Note: Currently, AutoStart is not qualified for use on any server that runs a firewall. However, if you are using a firewall, refer to the EMC AutoStart Administrator’s Guide for important guidelines.


https://support.emc.com/



AutoStart Domains

Domain networks

Each domain must have redundant networks so that AutoStart can detect outages and failures. There are multiple types of networks you can set up. It is important that all nodes in an AutoStart domain have access to the same networks. It is important that all nodes in the network have connectivity to the domain's primary agents.

These networks are:

◆ AutoStart domain networks, which are AutoStart’s main networks over which heartbeats are multicast, or point-to-point communicated, and includes communications for failing over a WAN

◆ Isolation detection networks

◆ Verification networks for the AutoStart domain

◆ EMC Symmetrix® networks for the AutoStart domain

Each type of network is described later in this chapter.

Note: Do not use DHCP for any node deployed for use with AutoStart. Nodes must have static IP addresses.

Planning a domain's networks

Make sure you understand how a domain’s application uses IP addresses. For example, does the application need a node alias? Does it need a NetBIOS name? Does it need an active computer name (ACN)? Does it need a virtual IP address with a DNS registration? All of this information is used by AutoStart to detect failures and relocate an application to a working node.

When planning your networks, make sure you:

◆ Identify all the nodes that need to be populated with an AutoStart agent, and that the nodes have unique names.

◆ For every domain line (every network you'll allow heartbeats on), make sure there is a source NIC or, more accurately (because NICs can have multiple IP addresses), source IP address for each server that has access to those networks.

◆ Have multiple connections to the networks, every domain network is connected to every node point-to-point.

◆ Can ping every node by fully-qualified domain name and by IP address.

◆ Can use multicast broadcasting or point-to-point communication, and that there is no point of failure along the path.

◆ Have redundant hardware where possible (for example, standby NICs). You can group NICs together and can test the responsiveness of the NIC, and can relocate managed IP addresses to a working NIC when there is a NIC failure. You can relocate the managed IP address to a physical, standby NIC.

◆ Verify a physical network connection for each network. If you have 3 networks, you must have network interfaces on each node. Do not have an interface that's installed but not connected, particularly if the driver for Windows is installed because

AutoStart domains 15

AutoStart Domains

AutoStart will try to use that NIC. You can disable any NIC that's in your server that you know you will not be using, although it would be better to remove it, though, if you're not going to use it.

◆ Must use multiple domain connections (or at least one heartbeat connection via the domain line, plus one verification line).

Multiple domains

You can have any number of AutoStart domains. Name each AutoStart domain so that you can easily identify it. All operations that AutoStart performs take place within the AutoStart domain. All management takes place within the AutoStart domain. All components of an application or tool that you're making highly available must reside inside the AutoStart domain. The nodes that highly-available applications and tools relocate to must also be part of the same AutoStart domain.

Note: AutoStart domains do not interact with each other; each is self-contained. For example, an application cannot relocate from one AutoStart domain to another.

Domain name format

You give an AutoStart domain its name at the time you install AutoStart on the first node in the domain. Observe the following guidelines when naming an AutoStart domain:

◆ Do not include hyphens (-) in a domain name.

◆ Domain names are case-sensitive. For example, Plant_57 and plant_57 are two separate AutoStart domains. EMC recommends that you use all lowercase characters in domain names, for consistency and simplicity.

◆ Once you create an AutoStart domain, you cannot change its name.

Domain tabs in the AutoStart console

An AutoStart domain’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, Licensing/Security tab, Statistics/Domain Failure Detection tab, Event Log tab, and Isolation Settings tab. You can see a list of all AutoStart domains by viewing the AutoStart domain table.

What to consider when working with an AutoStart domain

When working with an AutoStart domain, consider the following:

◆ Setting up the AutoStart domain — Make sure you complete the steps for setting up the AutoStart domain provided in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

◆ Proper naming of AutoStart domains — An AutoStart domain's name can have 1-8 alphanumeric characters. Each domain is named for identification purposes, and must be unique within your organization.

◆ Synchronization of the clocks on all nodes in AutoStart domains — AutoStart provides availability percentages for resource groups in the AutoStart domain (see the Availability Tracking tab for resource groups). AutoStart tracks events in the AutoStart


AutoStart Domains

database each node’s local time. Availability Tracking is most accurate when the clocks on all nodes in the AutoStart domain are synchronized. If the time difference between the AutoStart console and the node is large, the calculations may be skewed.

◆ Properly configure the Windows registry for mirroring — For more information, see AutoStart online help or the EMC AutoStart Administrator’s Guide.

You can view multiple AutoStart domains from any AutoStart console in the Windows domain or workgroup. The AutoStart domain that is installed on the node where you're using the console shows up automatically in the AutoStart console. To see other AutoStart domains, you must open them in the console.

Redundant networks in the domainWhen a resource group relocates, AutoStart moves all resources within the resource group from the current node to the specified node. All resources in a resource group relocate to the same node. Resource groups can only relocate to nodes specified in the preferred node list that have the software properly installed.

Note: In AutoStart documentation, the term relocation refers to any movement of a resource group or a resource from one node to another, and generally includes failover and failback, as well as a user's choice to move a resource group to another node for maintenance or to proactively prevent a potential problem.

Failover

Failover is the relocation of a resource group and its resources from the node where it typically runs to a backup node. As its name states, a failover typically means that a failure has caused the relocation of the resource group. Failover is sometimes called restart or migration.

Failover happens automatically when a node fails or resource fails and can't be restarted. You can also manually relocate a resource group prior to planned node outages or software upgrades on the current node.

You can configure resource groups to provide automatic failback of a resource group when the preferred node for the group comes back online after a failure. This functionality ensures that a resource group can always run on the node most suited for operation if that node is available.

Failback

The Auto Failback option, located on a resource group’s Options tab, causes the resource group to always try to go online on the first node in the preferred node list whenever that node is running. Select this option if you want the resource group to automatically relocate to the most preferred node whenever it is available.

Redundant networks in the domain 17

AutoStart Domains

HeartbeatsAutoStart uses a system of heartbeats to determine the state of nodes in a multicast AutoStart domain network. Once a node is online, the AutoStart agent on that node begins sending out heartbeats. Both primary and secondary agents send and receive multicast heartbeats on the same port number, but they use the heartbeat information differently:

◆ Primary agents receive heartbeats and use them for failure detection, as well as for isolation detection, by listening to the AutoStart domain for heartbeats from other nodes. Once a primary agent begins receiving heartbeats from another node, it expects to continue receiving them.

◆ Secondary agents use heartbeats only for isolation detection.

The interval at which the node sends out heartbeats is configurable, as well as the minimum detection time, the smallest interval which must pass without a heartbeat before the node is considered failed. By default, the heartbeat interval is one second, with a minimum detection time of fifteen seconds; you can change these values in the AutoStart console by using the domain's Statistics/Domain Failure Detection tab.

Additionally, when you are using the AutoStart for WAN for EMC SRDF® data source, integrity of the data can be enhanced by using the Symmetrix as an alternate verification network.

Note: Currently, AutoStart is not qualified for use on any server that runs a firewall. However, if you are using a firewall, refer to AutoStart online help or the EMC AutoStart Administrator’s Guide.

How AutoStart detects a failureAutoStart provides failure detection for an AutoStart domain's nodes, processes, and services. In the event of a failure of either a node or an application, AutoStart senses the failure and initiates recovery.

The purpose of failure detection is to:

◆ Prevent split brain (also called network partitioning) --- Split brain corrupts the data; you can recover from split brain only by restoring a saved copy of the data.

◆ Provide high availability — When there is a failure on a node that is running resources, the resources are no longer available.

Failure detection scenario

Here is an example of how failure detection works. You have two nodes:

◆ Active node A where the application is running

◆ Passive node B where the application can run if node A fails

Passive node B (the node that is waiting to launch if the active node fails) looks at heartbeats from node A. If heartbeats from node A cease, node B has to determine if node A has any connectivity, because if node A has any capability of reaching a client to modify attached data, it is imperative that passive node B not come online to take over for node A. How does it know not to take over? Here is how:


AutoStart Domains

1. If node B is missing heartbeats from node A, node B pings node A over the AutoStart domain network.

2. If it doesn't receive a ping back, node B pings node A over the verification network—that is, if a verification network is set up.

3. If it can't get a ping on those two networks, node B pings its own isolation IP address—that is, if an isolation address has been defined for the AutoStart domain or for the node—to find out if it (node B) has a connectivity issue itself. (The IP address used for isolation detection is a router or gateway or designated point that is always available on the network.)

4. If it can ping it's isolation address, then it knows that it (node B) is okay and that it is node A that is down or disconnected or nonfunctioning. At that point, the resource group or module instance relocates by coming online on node B.

So, having these redundant networks is what prevents split brain from occurring. Never run AutoStart with a single network. Will it work? Yes it will, but it creates a condition that can yield data corruption.

Networks and heartbeats

AutoStart uses a system of heartbeats to determine the state of nodes (via their AutoStart agents) in the AutoStart domain network. Each node sends heartbeats, and primary agents receive these heartbeats and use them for failure detection. There are four network communication options you can configure for determining the state of agents; you can configure them all to work together. They are:

◆ AutoStart domain network

◆ Verification network

◆ Isolation detection network

◆ Symmetrix network

You configure these node communication options on a node-by-node basis. How you do this is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide. Each of these networks is described in more detail later in this chapter.

Examples of how AutoStart detects a failure

Figure 1 illustrates a sample AutoStart domain configuration.

In this example, the company is set up with:

◆ Two 10 Mbit private networks (private AutoStart domain networks) on which the nodes exist. The two 10 Mbit networks are used as the AutoStart domain networks for the AutoStart domain, shown in the illustration as Private Domain 1 and Private Domain 2.

◆ One 100 Mbit local area network (LAN). This network is used as the verification network, shown in the illustration as Corporate LAN, a ping-only network.

◆ An AutoStart domain consisting of three nodes: two primary nodes and a secondary node.

How AutoStart detects a failure 19

AutoStart Domains

◆ Isolation addresses configured on an AutoStart domain-wide basis, using the IP address of the LAN’s router as the isolation address.

Figure 1 Sample AutoStart domain network configuration

Initially, all nodes are operating normally, with all agents sending heartbeats, primary agents receiving heartbeats for failure detection purposes, and secondary agents receiving heartbeats for isolation detection purposes. Heartbeat traffic is initially being sent using Private Domain 1.

Now, using the example illustrated here, take a look at the following examples of:

◆ A single network’s failure

◆ Dual network failure

◆ Total network failure

Example of a single network failureIf network communication is somehow lost for one network, such as if a network cable is unplugged, AutoStart responds by switching heartbeat communication to another AutoStart domain network, as shown in Figure 2.

Figure 2 Example of a single network failure

In this scenario, Primary Agent A, which both sends and receives heartbeats to determine failure, senses that it is no longer receiving heartbeats from Primary Agent B over the AutoStart domain network called Private Domain 1. The following takes place:

◆ Primary Agent A senses that it is no longer receiving heartbeats from Primary Agent B.

◆ Primary Agent B senses that it is no longer receiving heartbeats from Primary Agent A or the Secondary Agent.

◆ AutoStart switches over to using Private Domain 2 for heartbeat communication.


AutoStart Domains

Because there is no problem with Private Domain 2, heartbeats will be sent and received normally. AutoStart takes no further action.

Example of dual network failureIf network communication is somehow lost for both networks like the example in Figure 3, such as if both network cables are unplugged, each node in the AutoStart domain goes through a process to determine its state and other node states in the AutoStart domain.

Figure 3 Example of dual network failure

In this scenario, Primary Agent A, which both sends and receives heartbeats to determine failure, senses that it is no longer receiving heartbeats from Primary Agent B. The following takes place:

◆ Because it is still receiving heartbeats from the Secondary Agent, Primary Agent A knows that it is not isolated itself.

◆ Because it is not receiving heartbeats over either Private Domain 1 or Private Domain 2, Primary Agent A sends a ping to Primary Agent B over both the verification network and the AutoStart domain networks.

◆ The ping is not returned over the AutoStart domain networks, which have no connection, but is returned over the verification network, which is still connected. (The verification network is only used to determine if the node has completely failed. No heartbeats or data are sent over that network.)

◆ Because AutoStart knows that the node is running, Primary Agent A transitions the state of Primary Agent B to Agent Failed.

Meanwhile, Primary Agent B stops receiving heartbeats from both Primary Agent A and the secondary agent. The following takes place:

◆ After it has not received heartbeats from either node for half of the failure detection time, it attempts to see if it is isolated. It pings the router address and receives a response over the verification network. Therefore, Primary Agent B is not isolated.

◆ Primary Agent B pings Primary Agent A over the AutoStart domain networks and the verification network. Primary Agent B receives no response over the AutoStart domain networks, but does receive a response over the verification network.

◆ Because AutoStart knows that the node is running, Primary Agent B transitions the state of Primary Agent A (and the Secondary Agent) to Agent Failed. At this point, Primary Agent B forms its own AutoStart domain. To recover from this state, you repair

How AutoStart detects a failure 21

AutoStart Domains

the AutoStart domain networks and restart the AutoStart backbone process on one of the primary agents. In this case, it is recommended that the backbone process on Primary Agent B be restarted. To restart:

• On Windows nodes, the backbone process can be restarted from the command line or the Windows Services console.

• On UNIX nodes, the backbone process can be restarted by using:

$FT_DIR/bin/ft_Shutdown -backbone.

Example of total network failureIf network communication is somehow lost for both AutoStart domain networks as well as the verification network as shown in Figure 4, each node in the AutoStart domain goes through a process to determine its state and other node states in the AutoStart domain.

Figure 4 Example of total network failure

In this scenario, Primary Agent A (which both sends and receives heartbeats to determine failure) senses that it is no longer receiving heartbeats from Primary Agent B over Private Domain 1, which is handling all heartbeat traffic. The following takes place:

◆ Because Primary Agent A is still receiving heartbeats from the Secondary Agent, Primary Agent A knows that it is not isolated itself.

◆ Because Primary Agent A is not receiving heartbeats from Primary Agent B over the AutoStart domain network, Primary Agent A sends a ping to Primary Agent B over both the verification network and the AutoStart domain networks.

◆ The ping is not returned over any of the three networks.

◆ Because AutoStart gets no communication from node B, Primary Agent A assumes that the node itself has failed and transitions the state of Primary Agent B to Node Failed.

Meanwhile, Primary Agent B stops receiving heartbeats from both Primary Agent A and the secondary agent. The following takes place:

◆ After Primary Agent B has not received heartbeats from either node for half of the failure detection time, Primary Agent B attempts to see if it is isolated. It first sends a ping over the verification network and receives no response. Primary Agent B then sends a ping to the Isolation addresses and receives no response. There should be an Isolation address for each network the node is connected to.

◆ Primary Agent B determines that it is isolated and runs its isolation script, which forces the node to shut down, avoiding a network partition situation.


AutoStart Domains

◆ Primary Node B restarts, but its agent is unable to connect to the AutoStart domain because it is still isolated. After the isolation is removed, you must manually start the AutoStart backbone and AutoStart agent so that the isolated Primary Agent B can reconnect.

◆ Isolation detection takes place before failure detection occurs, so no node state changes take effect in Primary Agent B’s AutoStart database. When Primary Agent B fully initializes after a full reconnection, the AutoStart database on Primary Agent B then synchronizes with the other primary agents.

AutoStart domain networksHeartbeat communication within the AutoStart domain takes place over the AutoStart domain network; it is the primary way to determine if agents are online. In most environments, this network is used for AutoStart domain communication.

By default, heartbeats are sent out using a multicast over the AutoStart domain network and are received by every agent. As long as at least one primary agent is receiving heartbeats from the transmitting agent, the transmitting agent is listed in the Running state.

Note: Currently, AutoStart is not qualified for use on any server that runs a firewall. However, if you are using a firewall, refer to AutoStart online help or EMC AutoStart Administrator’s Guide for important guidelines.

Note: You can use the point-to-point communication option if multicast cannot be used. For more information, see “Multicast versus point-to-point communication” on page 24.

Network interfaces used for heartbeat communication are not managed by AutoStart. For that reason, do not use managed IP addresses for the network interfaces that you are using for heartbeat communication in an AutoStart domain network. Instead, use their base IP addresses.

You should set up redundant domain networks. Keep in mind that AutoStart sends heartbeats over all domain networks at all time, but AutoStart agents detect heartbeats using only one of those domain networks at a time. If AutoStart cannot detect heartbeats over one domain network, only then does it switch to the remaining domain network. If no domain network survives, then domain network communication ceases.

Note: It is possible to use IP over a Fibre Channel (such as a SAN). You can use your Fibre Channel as a routing mechanism for heartbeat traffic when, after setting up your domain network, you have only a single network connection over a fiber network remaining to the data. But you can still use ports over your fiber network to communicate heartbeats. This takes some additional configuration.

WAN configuration

The idea with a WAN configuration is to lets you run locally. If the local side of your WAN link becomes unavailable, you can fail across the WAN to the remote side. This provides you with a disaster recovery solution. To do this you must do the following:

AutoStart domain networks 23

AutoStart Domains

◆ Set up node groups and add a node group separator in the preferred nodes list of the resource group or module instance. For detailed information, see “Node groups” on page 98 and “Resource groups” on page 94.

◆ Crossing the WAN boundary to the remote nodes fails over to a completely different subnet. For that reason, you must make sure your clients must get updated DNS info so they can connect to the servers on the other side of the WAN link. To do this, you make the primary DNS server unavailable when there is a failover across a WAN link; this means that, in the host configuration on the clients, they would have to use the secondary DNS server; the secondary DNS server would have records that point to the other side of the WAN.

◆ Make sure you have:

• At least 2 primary agents on the local side of the WAN

• At least 2 more primary agents on the remote side of the WAN

• Multiple NICs and nodes on the remote side of the WAN, especially if the remote is your single point of failure.

◆ Make sure there are no single points of failure. Although a single point failure will be your WAN connection, make sure you have additional NICs in all servers, and additional gateways on each side of the WAN.

◆ Make sure you know which machines are capable of hosting the application or resource group resources. They must be similar in performance, hardware, memory, disk space on both sides of the WAN. On the remote side you must make sure you can run at the same service side.

Multicast versus point-to-point communication

For each node, you must specify whether it receives domain heartbeats over the AutoStart domain network via IP multicast sent from the domain's global multicast IP address, or via point-to-point communication:

◆ By default, heartbeats are sent out using an IP multicast over the AutoStart domain network and are received by every AutoStart agent. Multicast communication provides the broadest number of potential heartbeat recipients for the least overhead. Use multicast unless there is a specific need to use point-to-point communication in the AutoStart domain. Under normal circumstances, using multicast is more efficient.

◆ Point-to-point communication is most useful in a situation where the nodes in the AutoStart domain must communicate through a router or some other piece of network equipment that does not support IP multicast. If a device cannot forward the IP multicast, it may need to use point-to-point communication for failure detection between the machines that must communicate through the network equipment. Nodes that can communicate without using a router can communicate using either multicast or point-to-point.


AutoStart Domains

Isolation detectionIsolation detection is a mechanism that allows an AutoStart agent on a node to determine if its network communication has been cut off. The primary purpose of isolation detection is to protect against an outage or a situation in which a node cannot communicate with any resource, such as other nodes, routers, switches, and so on. The loss of communication typically occurs when a network card fails or a cable is removed.

To enable isolation detection, you must provide one or more IP addresses, called isolation addresses, to be pinged when the agent suspects that isolation may have occurred. We'll discuss these isolation addresses below, but first let’s talk about how isolation detection works.

Note: Currently, AutoStart is not qualified for use on any server that runs a firewall. However, if you are using a firewall, refer to AutoStart online help or the EMC AutoStart Administrator’s Guide for important guidelines.

Here's how it works

Heartbeats are sent out over the AutoStart domain network. These heartbeats are picked up by every primary agent. As long as at least one primary agent is receiving heartbeats from the transmitting agent, the transmitting agent is listed in the Running state.

Note: To prevent loss of network communication, use reliable, redundant networks. Isolation detection is not intended to provide total recovery in the event of a full network partition.

A primary agent suspects that isolation may have occurred if it receives no heartbeats for a period equal to half the minimum detection time (which is set on the AutoStart domain's Statistics/Domain Failure Detection tab). If this happens, it sends a ping to each user-defined isolation address:

◆ If at least one ping is returned, no action is taken, normal failure detection resumes.

◆ If the isolation detection ping fails, the agent assumes that its node is isolated; it enters isolation mode. Isolation mode prevents the node's agent from the following:

• Prevents it from assuming that all other nodes in the AutoStart domain have failed

• Prevents it from running rules and resource groups that are already running correctly on another node in the AutoStart domain

The node also invokes an isolation script to shut the node down (for more information, see “Isolation scripts” on page 27) and reboot it. Because isolation detection begins before the AutoStart domain detects the node's failure, the isolated node has a better chance of beginning its shutdown sequence before other nodes declare the node as failed. However, the act of detecting isolation and shutting down the node takes a finite amount of time, so it is possible that the node could be declared failed by the rest of the AutoStart domain before the shutdown sequence is complete.

Isolation detection 25

AutoStart Domains

After the node reboots (but before it fully initializes), the node’s agent verifies that the node is not isolated. If the agent finds that the node is still isolated, the agent waits, periodically pinging the isolation address before coming fully online. Manual intervention is required for the isolated agent to join back to the AutoStart domain—that is, you must manually restart the backbone and agent services.

Note: The agent does not automatically initialize and reconnect. However, the agent does initialize if the network connection had been restored before the system starts booting.

◆ If an agent determines that its node is not isolated and the minimum detection time is reached, failure occurs as normal.

Isolation addresses

You must specify the IP addresses to be used for isolation detection. Isolation addresses can be configured on an AutoStart domain-wide and a node-specific basis, however in most situations, domain-wide addresses are sufficient:

◆ Network isolation addresses (also called domain-wide or global isolation addresses) — Network isolation addresses are a set of common IP addresses that can be pinged from all nodes in the AutoStart domain. You specify a global isolation address in the AutoStart console by selecting the AutoStart domain, and then going to the domain's Isolation Settings tab. These IP addresses can include other machines on the same network, but should preferably include a router or hub to ensure that isolation is detected only when the node is completely isolated from the network. Regardless, the addresses you specify should be for devices on the network that are always in the AutoStart domain. They might include: primary nodes in the domain, the DNS server, firewall devices, a gateway, or any other device that is always available. You can specify up to 10 domain-wide, global isolation addresses. How you do this is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

◆ Node-specific isolation addresses (also called local isolation addresses) — You can specify a local address to be pinged by a particular node by using the node's Failure Detection and Mirroring tab. You might want to specify a local isolation address for a node that may not always have access to the devices listed as global isolation addresses. A node's local isolation addresses, if any are provided, are pinged in addition to the domain-wide, global isolation addresses specified for the AutoStart domain. You can specify an unlimited number of node-specific, local addresses. How you do this is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

If you don't set up isolation detection

Without isolation detection, an isolated agent assumes that all other nodes in the domain have failed; as a result, it initiates failover of all managed resources to itself, even though the resources are still running correctly on the other, unaffected node or nodes—this occurs only on a primary agent. In contrast, a secondary agent that becomes isolated immediately loses its connection to the backbone, and fails; from that point on, it is unable to acquire any resource. Meanwhile, the other nodes in the domain see the isolated node as failed, and start taking over any resources that were running on the affected node.


AutoStart Domains

If the isolated node is managing a shared disk resource it is important for the isolated node to release that disk resource before another node takes it over. This is the reason why isolation detection is required on a secondary node. The quickest and safest way to perform this action is for the isolation script to take over and shut down the isolated node as soon as possible. The actions of the other nodes to take over the resources of the isolated node may need to take into account the shutdown time of the isolated node.

Isolation scripts

An isolation script is a Perl script that is run if an active AutoStart agent detects that it has become isolated from the network (as described in “Isolation detection” on page 25). An agent runs the isolation script only when a node becomes isolated and the AutoStart domain or node has been configured with an isolation address.

An isolation script prescribes the actions to take place when an agent detects that its node has become isolated. An isolation script can react to the isolation in any manner, but because a node that is isolated from the network looks like a failed node to the rest of the AutoStart domain, EMC recommends that you make sure the isolation script does the following:

◆ Gives up the isolated node's resources, particularly shared disks

◆ Forces the isolated node to shut down

◆ Reboots the isolated machine

If the script is written to perform these tasks, the default behavior should cause the node to perform correctly.

Note: On primary agent nodes, intelligent shutdown scripts that use AutoStart’s API subroutines can also be used, allowing the node to be shut down gracefully. Because the primary agent maintains a rule interpreter, all rule API subroutines are available. The isolation script gets a parameter PRIMARY or SECONDARY which the script can then use to determine the agent type. To use the API subroutines for a script on a primary agent node, use require ft.pl;. For more information, see AutoStart online help or the EMC AutoStart Administrator’s Guide.

Use the domain's Isolation Settings tab to define the domain's isolation script.

Sample isolation script A default isolation script that flushes disk buffers to disk and then reboots the machine can be found in $FT_DIR\samples\IsolationScript. This sample script is also automatically included in the AutoStart database for your AutoStart domain, and appears as the default isolation script when you initially use the domain's Isolation Settings tab. It may look something like the following:

## This script will then be run when a node becomes isolated if you have# defined some isolation addresses under Failure Detection Settings.# This script will be called with 1 parameter -# PRIMARY - if it is running on a primary node# SECONDARY - if it is running on a secondary node### Determine what applications are running on this node and shut them down# as quickly as possible because they will be migrated to a node that# is not isolated.#

Isolation detection 27

AutoStart Domains

# if($ARGV[0] eq "PRIMARY") {## If the script is running on a PRIMARY then it has complete access to all of# the ft:: perl functions. These can be used to determine what resource groups# are running on this node.# The next 2 lines give you access to the ft:: functions.## push (@INC, "$ENV{$_DIR_ENV_VAR_}/bin");# require "ft.pl";## &ft::PostEvent($ft::SEV_ERROR,"$_PRODUCT_NAME_ Isolated, shutting system down");## }elsif($ARGV[0] eq "SECONDARY") {## If the script is running on a SECONDARY then it does NOT have access to any# of the ft:: functions and must use other means to determine what processes# are running on this node.## }require "$ENV{FT_DIR}/bin/product.pl";require "$ENV{FT_DIR}/bin/ft.pm";&ft::ft_GetOSType($osType);if($osType eq "WIN_NT") {`$ENV{$_DIR_ENV_VAR_}/bin/logevent.exe -s E "$_PRODUCT_NAME_ Isolated, shutting system down"`;}elsif ($osType eq "SOLARIS") {ècho "$_PRODUCT_NAME_ Isolated, shutting system down" > /dev/console`;}elsif($osType eq "HPUX") {ècho "$_PRODUCT_NAME_ Isolated, shutting system down" > /dev/console`;}elsif($osType eq "AIXRS") {ècho "$_PRODUCT_NAME_ Isolated, shutting system down" > /dev/console`;}elsif($osType eq "LINUX") {ècho "$_PRODUCT_NAME_ Isolated, shutting system down" > /dev/console`;}# # Shut the system down as quickly as we can.#if($osType eq "WIN_NT") {`$ENV{$_DIR_ENV_VAR_}/bin/sync.exe`;`$ENV{$_DIR_ENV_VAR_}/bin/sync.exe`;`$ENV{$_DIR_ENV_VAR_}/bin/shutdown.exe /L /R /T:0 /Y /C`;}elsif ($osType eq "SOLARIS") {`/bin/sync`;`/bin/sync`;`/etc/reboot`;}elsif($osType eq "LINUX") {`/bin/sync`;`/bin/sync`;`/sbin/reboot`;}elsif($osType eq "HPUX") {`/usr/sbin/sync`;`/usr/sbin/sync`;`/usr/sbin/reboot`;}elsif($osType eq "AIXRS") {`/usr/sbin/sync`;`/usr/sbin/sync`;`/usr/sbin/reboot -q`;}


AutoStart Domains

Verification networks for the AutoStart domainThe verification network is an optional, secondary route that can be configured on a node-by-node basis. The verification network is invoked only if no heartbeats are received over the AutoStart domain network. It is used only as a secondary means to ping a node to determine if the agent is failing or if the node itself is failing. Standard heartbeats and normal communication never use the verification network. It is used only to send a ping if the AutoStart domain network’s ping fails.

Note: Currently, AutoStart is not qualified for use on any server that runs a firewall. However, if you are using a firewall, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for limitations.

If an AutoStart domain has a verification network, failure detection performs an extra check before declaring the node or agent failed. When heartbeats are no longer received from a node for the duration of the Minimum Detection Time, the usual ping is sent using the AutoStart domain network. If that ping fails, an additional ping is sent to the node using the verification network line. Like the ping sent over the AutoStart domain network, if the ping is:

◆ Returned, the node's state transitions to Agent Failed.

◆ Not returned, the node's state transitions to Failed.

Note: The AutoStart console displays icons for each resource; these icons alert you to the resource’s condition. When a resource has gone offline or failed, the status icon turns red, like this: .

Additionally, integrity can be enhanced by using the EMC Symmetrix as an alternate verification network. If an event occurs requiring verification checks, AutoStart will verify the health of other hosts via direct connections to a Symmetrix or through the SRDF links.

Symmetrix networks for the AutoStart domainAdditionally, integrity can be enhanced by using the EMC Symmetrix as an alternate verification network. If an event occurs requiring verification checks, AutoStart will verify the health of other hosts via direct connections to a Symmetrix or through the SRDF links.

When you are using the AutoStart for WAN for SRDF data source, integrity of the data can be enhanced by using the Symmetrix system as an alternate verification network. If an event occurs requiring verification checks, AutoStart will verify the health of other hosts via direct connections to a Symmetrix system or through the SRDF links through the use of the Symmetrix HeartBeat Table. EMC SRDF Heartbeating, used with the AutoStart for WAN for SRDF, will only work with Primary agents that are connected directly to the Symmetrix system. Refer to EMC documentation for more information on Symmetrix HeartBeat Tables.

Note: Attempting to configure Symmetrix Heartbeats on a node where Solutions Enabler is installed, but no Symmetrix devices are present, will result in an extended blocking operation that may last several minutes.

Verification networks for the AutoStart domain 29

AutoStart Domains

For more about setting up Symmetrix network for an AutoStart domain, see AutoStart online help or the EMC AutoStart Administrator’s Guide.

Mirroring the Windows Registry MonitorThe Windows Registry Monitor mirroring utility runs on AutoStart and its supporting Windows operating systems. Registry mirroring is a process that continuously monitors and saves registry data into any available file system data source, so that after a failover or relocation, registry content remains accurate and up-to-date.

Why use AutoStart's Registry mirroring?

Many Windows applications use the Windows Registry Monitor (also called the Registry) to access information needed for their proper execution. Information stored in the registry may be dynamic and changing, based on the operation of the application or other applications.

Over time, the values of the registry keys that the application accesses may be different from the information that was stored in the keys when the application was installed, when the machine was initialized, or when the application was launched. If the machine or the application then fails, and AutoStart relocates the application to another node, the information stored in the registry on the failover node may be wrong or out of date. AutoStart Registry mirroring replicates the registry keys that you specify and keeps this the registry’s keys accurate and up-to-date. An application may require these updated entries to start running on the backup host as if no failure had taken place. If a node or application fails, the most recent values of the monitored registry keys from the production host can be written to the registry on the backup host, so that the application accesses the latest information.

The processes

Registry mirroring requires the following executables which AutoStart installs in the FT_DIR\bin directory:

◆ RegMon2.exe — Registry mirroring utility that functions as an AutoStart process

◆ RegCheckpoint.exe — Registry mirroring checkpoint utility that functions as an AutoStart utility process

How AutoStart's registry mirroring works

Registry mirroring is done by an AutoStart process. Multiple registry keys can be monitored with a single process. You configure the AutoStart Registry mirroring process using the AutoStart console. Once the process is configured, you include the process in a resource group along with the application that requires access to the registry.

During operation, the process monitors activity to the registry and if the specified registry values change, the new values are written to the configured location in the file system data source. If values for the specified registry keys change, the new values are exported to files on the resource group's file system data source, where they are stored. If the resource group relocates, the registry entries that are needed by the application are


AutoStart Domains

automatically updated to allow the application to begin running on the new node as if no failure had taken place; the exported registry entries are put to use to establish their up-to-date values on the node where the resource group is being relocated to.

Note: As with all operations that modify the Windows registry, only use the Registry mirroring utility if you are experienced with using and changing entries in the Windows Registry Monitor. Do not edit the registry unless it is absolutely necessary. Entering erroneous entries in the registry may cause the node to crash or cause applications to malfunction.

When the resource group is brought online, registry mirroring writes output to a list of files. The .regdat files contain the current values of the monitored Windows registry keys. If the value of a key changes, then the old key value is written to a .regdat.prev file. The new key value is written to the .regdat file. When the registry mirroring process imports key values into a registry after a failover, it starts by importing the content of the .regdat.prev files, then it imports the .regdat files. This assures that a value is written if a failover occurred during a write to these files.

When a key is deleted, the .regdat file is renamed to .regdat.archive. After a failover, the registry key on that node will also be deleted.

Mirroring methods

For each key that you mirror, you can specify one of the following methods for mirroring it:

◆ Replace the entire key’s contents. The key's content is deleted and all the data in the registry key is replaced with the saved data. The exported registry data file is renamed.

◆ Merge the registry key with the data stored in the data source. Unlike in the Replace method, changes are merged with existing values in existing keys, or new keys are added.

◆ Delete the registry key and rename the exported registry data file with an added .archive extension.

Registry checkpoint

The Registry checkpoint is an AutoStart utility process used in resource groups along with the Registry mirroring process. The checkpoint blocks the execution of registry operations until the registry completes import operations during the resource group's startup sequence. The checkpoint is similar to a delay in that it waits for the Registry mirroring process to complete its import operations before allowing the startup sequence to proceed.

Set up and use

It is assumed that one instance of the Registry mirror monitors all of the Registry keys for one resource group. If there are multiple resource groups in an AutoStart domain, you can set up a different Registry mirror instance for each resource group.

To set up and use an instance of Registry mirroring, follow the steps that are provided in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

Mirroring the Windows Registry Monitor 31

AutoStart Domains

User access and securityA user must have access to the AutoStart domain to be allowed to start an agent, run the command line interface (CLI), or launch the AutoStart console. A user must have administrator-level rights to be able to give users access to an AutoStart domain.

◆ Installations on Windows add the user who installs the software as the initial user in the database.

◆ Installations on UNIX include only the root account as a valid user in the AutoStart database.

Initial administrator for the AutoStart domain

The AutoStart domain is created when a user installs the first agent on a node in the AutoStart domain. (Agent installation is described in the EMC AutoStart Installation Guide.) The user who installs the first agent is automatically added to the AutoStart domain as a user, and is automatically assigned a level of Administrator for:

◆ The Windows domain (for an agent installed on a node in a Windows domain)

◆ The node itself (for an agent installed on a node in a Windows workgroup)

◆ The root account (for an agent installed on a UNIX node)

That initial administrator can then define additional user accounts for the AutoStart domain, including additional administrators. Only administrators can define additional users.

Levels of security

There are three levels of user security. Note that a user's security level can differ by node or by Windows domain. Levels are:

◆ Administrator

◆ Operator

◆ User

Note: Any user that is not defined as an Administrator must have write permissions to the AutoStart console directory: InstallDir\consoleReleaseNumber. This is required so that the user-level or operator-level user can see all of the AutoStart modules in the AutoStart console.

Administrator level

An administrator-level user (also called an administrator) can define, delete, and modify objects in the AutoStart domain. Give administrator-level access to users who need access to everything in the AutoStart domain, including the ability to:

◆ Grant users access to AutoStart

◆ Enter license keys

◆ Define resource groups and other objects in the AutoStart console


AutoStart Domains

◆ Generally define and refine your application definitions to AutoStart

◆ Perform operator-level and user-level actions (described below)

◆ Access Advanced user mode (see AutoStart online help or the EMC AutoStart Administrator’s Guide for details.)

Note: There must always be at least one user with Administrator access to the AutoStart domain.

Operator level

An operator-level user (also called an operator) can view and manage objects in the AutoStart console, but cannot modify or delete objects in the AutoStart domain. Grant operator-level access to users who need to react to AutoStart conditions by relocating resource groups, and starting and stopping processes, in addition to performing user-level actions.

Note: While the operator-level user can list all objects in AutoStart console, there are some exceptions to LDAP related commands, which can only be accessed by Adminstrator-level user. The commands listLocalDirUsers, listLdapGroups and getLdapInfo will be denied access to users with PERM_OPER security level.

User level

A user-level user can view objects in the AutoStart domain, but cannot execute, modify, or delete objects. Give user-level access to users who need to access the event log, and see objects in the AutoStart console. User-level users cannot define, modify, or delete objects in AutoStart, nor can they start and stop services and relocate resource groups.

Access from locations

When defining user access to the AutoStart domain, you specify the location or locations a user can log in from and you can vary the user's access (Administrator, Operator, or User) depending on the location. You may need to add the user multiple times to account for all of the ways in which the user will log in. You can give access to the user logged in to:

◆ A specific local node account; you can add the user for each node account the user is valid for.

◆ A specific Windows domain account; you can add a user for multiple domains.

◆ Any node or Windows domain account.

◆ The local Windows workgroup.

Additional, underlying OS-level rights

Always remember that, depending on the configuration of an AutoStart domain, and depending on the AutoStart modules, applications, or tools you are using in the domain, there may be additional security restrictions for sites or the operating system (OS). For example, when you use AutoStart to create services, processes, node proxies, state

User access and security 33

AutoStart Domains

monitors, and AutoStart modules, you will be prompted for OS-level login information, and the account you provide must have the appropriate access to complete the operation to be performed. Other examples include:

◆ If you are using the IIS module, there are security measures outlined in AutoStart online help and the EMC AutoStart Administrator’s Guide, when you configure your IIS sites.

◆ If you are using the Registry Monitoring tool, there may be security requirements for the user account used for monitoring the registry too, as described in “Mirroring the Windows Registry Monitor” on page 30.

◆ If you are using a Telnet utility process, there are authentication requirements when you use the Telnet utility process Settings tab.

◆ Any custom or user-defined resource groups may require their own access restrictions.

Using definition files with AutoStartUse ASCII text-based definition files to hold definitions of objects for the AutoStart domain. Object definitions include:

◆ Resource definitions

◆ Sensor, trigger, and actuator definitions

◆ Rule text

All database backups are written to definition files; database restoration imports the contents of the backup definition file. Likewise, you can export the AutoStart database to a file for the purpose of making modifications or importing objects to another AutoStart domain. You can even modify the definition files before re-importing them or importing them into another domain.


CHAPTER 2Nodes and Their Agents, NICs, and IPs

This chapter describes nodes in the AutoStart domain, and the agents that run on them:

◆ Nodes..................................................................................................................... 36◆ AutoStart agents ..................................................................................................... 36◆ NICs........................................................................................................................ 37◆ NIC testing .............................................................................................................. 39◆ When a NIC fails...................................................................................................... 41◆ Usable, standby, and reserved NICs ........................................................................ 44◆ NIC groups .............................................................................................................. 44◆ IP addresses ........................................................................................................... 46◆ Node aliases for Windows ....................................................................................... 49◆ Node proxies........................................................................................................... 52

Nodes and Their Agents, NICs, and IPs 35

Nodes and Their Agents, NICs, and IPs

NodesThe node is the basic building block within an AutoStart domain. A node is any machine with an AutoStart agent installed and running. Applications that are managed are installed on the node. Other non-managed applications can also be running on the node.

The AutoStart agent provides the monitoring and management capabilities within the node. There are two types of AutoStart agents: “Primary agents” and “Secondary agents”. The node's configuration determines which type of agent is on the node.

Node tabs in the AutoStart console

A node’s AutoStart information is provided in the following tabs in the AutoStart console:

◆ The Properties tab

◆ The Failure Detection and Mirroring tab

◆ The Agent Statistics tab.

You can see a list of all of an AutoStart domain's nodes by viewing the node table.

AutoStart agentsA primary agent is a node running an agent, process monitor, rule interpreter, and state monitor, which provides the main activities for managing an application. A primary agent also maintains a copy of the AutoStart database. At least one primary agent must be running at all times for an AutoStart domain to function properly. There are typically two to five primary agents in an AutoStart domain. Any AutoStart domain of two or more nodes should include at least two primary agent nodes. Other nodes in an AutoStart domain should run as secondary agents. You can see a summary of information about a node’s agent by viewing the node’s Agent Statistics tab.

Primary agents

A primary agent manages services and processes, executes rules, manages resource groups, monitors the state of nodes and the processes running on nodes, and maintains the AutoStart database. Each primary agent includes the following components:

◆ The AutoStart backbone process

◆ The AutoStart agent process

◆ The process monitor

◆ A rule interpreter

◆ The AutoStart database

◆ A state monitor

The backbone process provides communication for the entire AutoStart domain. The agent process, process monitor, and rule interpreter form the events and rule interpreter, which manages and monitors rules and resource groups in the AutoStart domain.



The AutoStart databaseThe AutoStart database maintains information, such as resource definitions and states, on each primary agent. It is replicated to each primary agent in the AutoStart domain. Whenever a change occurs in the AutoStart domain, the AutoStart agent recognizes the change then synchronizes the replicated databases by sending the information to each copy of the database, maintaining the replica synchronization. As long as there is at least one copy of the database available, management and rule evaluation continues.

If you have a situation in which you want to duplicate the AutoStart domain information to create a new AutoStart domain, you can export or import the AutoStart database.

Secondary agents

A secondary agent provides the same management and monitoring capabilities as a primary agent except for the following:

◆ It does not run a backbone process, although it is connected to the backbone that runs on the primary agent.

◆ It does not execute rules.

◆ It does not maintain a copy of the AutoStart database.

Each secondary agent includes the following components:

◆ The agent process

◆ The process monitor

◆ A state monitor

You can promote a secondary agent to be a primary agent.

NICsA network interface card (NIC) provides the means for a node to communicate across a network. You are going to set up AutoStart so that if a NIC fails, another NIC can take over for it.

Note: It is recommended that there be at least three NICs available on the node if you are using NIC-to-NIC failover. Having three interfaces gives you access to a domain network plus a redundant network, as well as a NIC to be used for failover.

Each NIC must have a base IP address; once a NIC has a base IP address, network communication can take place using that address. However, for AutoStart to manage the NIC, you must also give it a managed IP address so that AutoStart can monitor and react to the NIC’s behavior. A managed IP address acts as a virtual IP address that AutoStart can move from NIC to NIC in case of a failure; it relocates with the application should AutoStart need to relocate the application on another node. For more information, see “IP addresses” on page 46.

AutoStart groups NICs into like configurations (called NIC groups), and tests the responsiveness of NICs to make sure they are alive. If a NIC fails, AutoStart can transfer the NIC’s IP addresses to a working NIC. You can set up your NIC groups so that if a NIC fails, its IP addresses are relocated to an available NIC. For details, see “When a NIC fails” on

NICs 37


page 41.

Note: NIC functionality may vary depending on the brand of card, the platform, and the version being used.

You can manage NIC behavior and configuration by grouping NICs to provide like configuration across NICs, testing the responsiveness of the NIC to determine if it can be used, and allowing IP addresses to be failed over to a standby NIC in case of failure.

AutoStart detects NICs

When an AutoStart agent starts up, it detects each NIC in the AutoStart domain. AutoStart detects the NIC’s base IP address, and uses the base IP address as the NIC’s ID. When AutoStart initially detects a NIC, it sets the card's state and usage to the following default values:

◆ It sets the state (which determines its general availability for use) to Alive.

◆ It sets the usage (which determines how AutoStart can use the NIC) to Usable. You can change the card’s usage to Standby or to Reserved. For details about what each usage means, see “Usable, standby, and reserved NICs” on page 44.

NIC group

When an AutoStart agent starts up and detects a NIC for the first time—either because the node had been added to the AutoStart domain or because the NIC has been added to a known node—it adds the NIC to a NIC group that uses the same subnet. (AutoStart takes the NIC’s base IP address and, from that, derives its subnet. It then adds the NIC to the NIC group of the same subnet.) If necessary, you can reconfigure a NIC into a different NIC group after the initial detection process is complete. For more information, see “NIC groups” on page 44.

Unplumbed NICs

On a UNIX node, a NIC can become unplumbed if AutoStart fails over the NIC’s IP addresses to another NIC on the node. If a NIC becomes unplumbed but becomes available again, you can reset it in AutoStart. (On Solaris nodes, unplumbed NICs are always tested so there is no need to reset a unplumbed NIC on a Solaris node.) For more about unplumbed NICs, see “NIC-to-NIC failover” on page 42.

An unplumbed NIC cannot be managed by AutoStart because, by definition, it has no IP addresses assigned to it. AutoStart does not automatically include unplumbed NICs in a NIC group. This is because an unplumbed NIC has no base IP address that AutoStart can use to assign it to a NIC group. However, you can add an unplumbed NIC to any NIC group, so that AutoStart will at least test the unplumbed NIC to make sure it is alive. For optimum performance, make sure you add the NIC to a NIC group that uses the same subnet.

Checklist for setting up NICs

After you have determined what your AutoStart domains are, and have installed the agents, now you need to set up your NICs:

1. Identify the NICs in the AutoStart domain.



2. Make sure each NIC is assigned a base IP address. Plan the NICs' base IP addresses with their subnet paths in mind, keeping in mind that NIC-to-NIC failover requires that the NICs share a subnet address and must be in the same NIC group.

3. Include any NICs that do not show up in AutoStart; exclude any NICs that you do not want available to AutoStart.

4. Arrange the NICs into groups that share a subnet and share testing addresses and frequencies.

5. Identify the group's Standby NICs. you have to have standby NICs to activate the NIC to NIC Failover option.

6. Set up Usable NICs for NIC-to-NIC failover. (Alternatively, use IPMP (IP network multipathing).)

7. Create managed IP addresses and network paths.

Each of these steps is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

Viewing NICs in the AutoStart console

All NICs are listed in the AutoStart console. You can view the current usage of each NIC from the AutoStart console. In the tree view of the AutoStart console, a NIC is identified by its node and interface (for example, as “NodeA: Local Area Connection”). You can easily identify untested NICs, unplumbed NICs, and NICs on Windows nodes in a NIC group without any configured test IP addresses because they are identified in the AutoStart console with a different icon than the other NICs.

If a NIC does not show up in the AutoStart console, you can take steps to include it in AutoStart; or if it is necessary that a NIC not be recognized by AutoStart, you can exclude it.

NIC tabs in the AutoStart console

A NIC’s AutoStart information is provided in the Settings tab in the AutoStart console. You can see a list of all NICs by viewing the NIC table.

NIC testingOnce a NIC is detected in the NIC group, AutoStart begins testing the NIC. AutoStart tests NICs by pinging a list of IP addresses. If AutoStart can ping a NIC, the NIC is usable and there is no need to relocate its managed IPs to a standby NIC.

All detected NICs are tested in accordance with the testing options defined for the NIC’s group. Each NIC in a NIC group is on the same physical subnet, and each shares the same set of test addresses and test frequency. You can't test a NIC that’s not in a NIC group, so make sure all NICs are in a NIC group. For more information, see “NIC groups” on page 44.

You must manually configure the test IP addresses of a NIC group. Once configured, all NICs in the group use the same test IP addresses.

To prevent a plumbed NIC from being tested, remove it from the NIC group and set its usage to Reserved. This prevents the NIC from being automatically placed in the group when the agent restarts.

NIC testing 39


Testing on UNIX

On UNIX, a default test uses network traffic as an indicator of viability, but a more robust test can be configured for each NIC group. For more information, refer to “When a NIC fails” on page 41.

On UNIX nodes, by default the testing behavior is to periodically check to see if there has been any recent traffic on the network. Because the default heartbeat mechanism sends Multicast messages every second, this network test should provide a reliable means of checking a NIC’s responsiveness for both plumbed and unplumbed NICs.

Note: Linux nodes: Unplumbed NICs on Linux nodes are not tested.

If no recent activity is detected, a more robust second level test is performed using IP addresses and a system of pings to determine if the NIC has failed. For this test to work correctly, the user must supply in the NIC group configuration the list of test IP addresses to be used. It is strongly recommended that these test IP addresses be configured after the NIC group is created to provide maximum reliability when testing NICs. If no IP addresses are supplied, the subnet broadcast address is used by default except on Linux systems where 224.0.0.1 is used by default.

Up to ten test IP addresses can be defined (for optimal efficiency, use only two or three). Test addresses are tested sequentially at the specified interval. If a ping is returned from the first test address, it is assumed that the NIC is performing normally, and no further testing is performed until the next interval. If the ping times out, the next test address in the list is tested. If all IP addresses fail, each IP address is retested up to the number specified in Test Repetitions. If, after all the repetitions have completed, there is still no response, the NIC is transitioned to the No Response state.

Configuring routers. If the Failure Detection Settings have been modified so that Multicast messages are not used, and no nodes are using the specified network, unplumbed NIC testing may exhibit unusual behavior. To ensure reliable results for unplumbed NIC testing when not using multicast on a particular network, make sure your routers are configured to pass Multicast messages.

Testing on Windows

On Windows, the ping test for IP addresses is the only supported test. Although you can define up to ten test IP addresses, for optimal efficiency use only two or three. Test addresses are tested sequentially at the specified interval. If a ping is returned from the first test address, it is assumed that the NIC is performing normally, and no further testing is performed until the next interval. If the ping times out, the next test address in the list is tested. If all IP addresses fail, each IP address is retested up to the number specified in Test Repetitions. If, after all the repetitions have completed, there is still no response, the NIC is transitioned to the No Response state.

Custom NIC Testing

On UNIX, custom NIC tests can be run through the use of the FT_NIC NIC_Name environment variable. Place the custom NIC test script in the $FT_DIR/bin directory and set the FT_NIC environment variable to the name of the script. The script must return a 0 for success and a nonzero for failure.



To enable a custom NIC test for just LAN1, place the script in the $FT_DIR/bin directory and set the FT_NIC_lan1 environment variable to the name of the script.

To have a NIC always pass a NIC test, set the environment variable to SUCCESS.

For example, if the test interval for the NIC is 30 seconds, the Test IP Timeout value is 10 seconds, the Test Repetition value is 2 and there are three test IP addresses defined, the test is performed as shown in Figure 1 on page 41.

In this scenario, 90 seconds elapse before the NIC transitions to the No Response state. Therefore, a balance should be struck between the number of configured test IP addresses and the number of test repetitions. Typically, if more than two test IP addresses are configured, the need for test repetitions is minimized and should be used only in extreme cases.

Figure 1 Test timeout

When a NIC fails

Note: What happens when a NIC fails depends on the settings you have configured for the card and for the card’s NIC group. This topic describes the behavior for various settings. All descriptions below assume that IP addresses are assigned to the NICs, and the NICs belong to a NIC group that is tested. If a NIC is not included in a NIC group, AutoStart does not monitor it, nor does it relocate the NIC’s IP addresses if the NIC fails. See “NIC groups” on page 44.

When a NIC fails 41


When a NIC belongs to a NIC group, AutoStart assumes responsibility for all IP addresses on the NIC. The IP addresses that can reside on a NIC are:

◆ Managed IP addresses — IP addresses assigned to the NIC through AutoStart. Managed IP addresses may be virtual if the IP address is moved only by AutoStart or physical if the IP address is the main IP address assigned to a NIC, but AutoStart’s behavior is identical for each.

◆ Unmanaged IP addresses — IP addresses assigned to the NIC through a means outside of AutoStart.

Assuming that at least one of a NIC’s IP addresses is being managed via a managed IP address in AutoStart, if the NIC fails:

◆ The ft_NicState sensor changes the NIC’s state to No Response. You can create and enable rules that monitor the ft_NicState sensor and react to a failure.

◆ All managed IP addresses on the NIC are transitioned to the Path Failed state. This updates the ft_IP_AddressState sensor value, causing any associated triggers for resource groups and rules to fire and relocate the managed IP address.

◆ If the NIC's NIC-to-NIC failover option is:

• Not enabled --- AutoStart leaves the NIC’s IP addresses on the NIC, even though they are unavailable.

• Enabled --- refer to the “NIC-to-NIC failover” discussion, next in this topic.

NIC-to-NIC failover

If all IP addresses, including the base IP address, are removed leaving the card unplumbed, the NIC must be reset manually once the NIC is once again alive and usable.

The behavior that occurs when a NIC fails with NIC-to-NIC failover configured is described below:

◆ Unmanaged IP addresses on UNIX nodes — Unmanaged IP addresses are moved to the Standby NIC by way of NIC-to-NIC failover. If the NIC-to-NIC failover procedure fails, the unmanaged IP address may be left in an unassigned state. The ft_NicState sensor value is updated.

◆ Unmanaged IP addresses on Windows nodes — Unmanaged IP addresses are not moved to the Standby NIC. Windows maintains the IP address on the NIC, and it will be assigned to the NIC if it becomes available again. The ft_NicState sensor value is updated.

◆ Managed IP addresses — IP addresses are moved to the Standby NIC, and the AutoStart database is updated with new NIC information. If the NIC-to-NIC failover procedure fails, the IP addresses are marked as unavailable. In a resource group or rule, a trigger will be fired to move the IP address to another node in the AutoStart domain. The ft_NicState sensor is activated. The ft_IP_StateSensor is not updated, so AutoStart reacts as if no IP failure occurred.

Note: NIC functionality may vary depending on the brand of card, the platform, and the version being used.



If a Usable NIC fails, all managed IP addresses of the failed NIC relocate to a Standby NIC. The usage of the Standby NIC transitions to Usable, and the unavailable NIC’s usage transitions to Standby. Which standby NIC the IP addresses are failed over to is determined by how you configured the NICs:

◆ If you configure the failed NIC so that its managed IP addresses relocate to a specific NIC, all of the IP addresses from the failed NIC are moved to the specified failover NIC.

◆ If you configure the failed NIC to Auto Select, a NIC is chosen from the list of NICs in the NIC group whose current usage is Standby.

How you set this option is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

On WindowsOn a Windows node, only the managed IP addresses of a failed NIC relocate to another NIC on the node, provided that the cards are configured with the correct NIC usages; base IP addresses and unmanaged IP addresses do not relocate.

Before you can use NIC-to-NIC failover for a NIC on a Windows node, the NIC must be configured with a base IP address. You can configure the base IP address using the Network applet of the Windows Network Control Panel. Due to operating system constraints, the base IP address is permanent and immovable and AutoStart cannot fail it over.

If a NIC fails, its base IP address and its assigned, unmanaged IP addresses become unassigned; they must not be reassigned to another NIC. Even though you cannot view the IP address status of the failed NIC, the operating system maintains that the IP address is still assigned to the NIC; when the NIC becomes available again, those IP addresses are listed as assigned to the NIC.

On UNIXNIC-to-NIC failover on UNIX nodes enables failover of all managed and unmanaged IP addresses from the failed card to another card on the node provided that the cards are configured with the correct NIC usages, which is the following: one card’s usage must be set to Usable and one or more cards must be set to Standby. If the failed NIC was configured to:

◆ Relocate to a specific NIC, all IP addresses are moved from the failed card to the specified card.

◆ Auto Select, all IP addresses are moved from the failed card to a card in the NIC group whose current usage is Standby.

Note: On Solaris, unplumbing or downing a NIC will not cause it to be recognized as failed. To test NIC-to-NIC failover on Solaris, the network cable must be unplugged from the NIC. Unplugging the cable will cause the NIC to be reported as failed, and initiate a NIC-to-NIC failover operation.

Note: On Solaris, instead of using NIC-to-NIC failover, you can use IPMP. For more information, refer to the EMC AutoStart Administrator’s Guide or AutoStart online help.

When a NIC fails 43


Usable, standby, and reserved NICsThe NIC’s usage determines if it can be used for NIC-to-NIC failover, as well as determining if the NIC can be used for managed IP address assignment in general. To view the current usage, select the NIC in the tree view of the AutoStart console, which displays the NIC’s Settings tab. How you specify NIC usage is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

The possible usages for NICs are:

◆ Usable — The NIC is available for IP assignment. A Usable NIC can be assigned IP addresses, and may be used if its status is Alive at the time of IP address assignment.

◆ Standby — The NIC is designated as the destination NIC for network card failover, and is not currently in use. AutoStart will not assign managed IP addresses to the NIC except during NIC-to-NIC failover. A Standby NIC is used only to receive IP addresses during NIC-to-NIC failover. Standby NICs are not used for general IP address assignment.

Note: If you are using NIC-to-NIC failover, the cards you select must use a combination of the Usable and Standby usages.

◆ Reserved — AutoStart tests this NIC for response if the NIC is included in a NIC group, but AutoStart does not use the NIC for failover, and will not assign managed IP addresses to the NIC. However, you can explicitly assign managed IP addresses to the NIC. AutoStart does not automatically include a NIC whose usage is Reserved in a NIC group. When you set a NIC’s usage to Reserved, you are setting the NIC aside for a specific IP address assignment configuration. Therefore it is not used for standard IP address assignment.

For more information on designating a specific NIC for IP address assignment, see “Node aliases for Windows” on page 49.

NIC groups A NIC group is a set of NICs in the AutoStart domain. Its purpose is to provide a way for AutoStart to organize and manage the availability testing of a set of NICs on common subnet:

◆ Each NIC in the group shares the same set of test addresses and test frequency. Rather than specifying separate test options for each NIC, you specify one set of tests for all the NICs in the NIC group.

◆ Each NIC in the group must be on the same physical subnet.

◆ The NIC group also identifies any NICs you want held on standby. If a NIC in the group fails, AutoStart can relocate IP addresses from the failed NIC to a standby NIC that is alive.



Auto-creation of a NIC group

AutoStart creates NIC groups automatically when the agent starts up. When an agent starts up and detects a NIC for the first time-either because the NIC’s node has been added to the AutoStart domain, or because the NIC has been added to a known node-it adds the NIC to a NIC group. Using the base IP address of the NIC—NIC groups are organized by subnet-AutoStart automatically adds the NIC to the NIC group for its subnet.

AutoStart creates the NIC group if it isn't already created, using the group’s subnet address as the NIC group’s name. You can reconfigure NIC groups after the initial detection process is complete. You can even rename NIC group. Changes to the NIC group are propagated immediately to all nodes where NIC testing is taking place for the group.

Note: The first time an AutoStart agent starts up, it adds all the NICs in a subnet to one NIC group. You can, however, break up the NIC group to satisfy how you need to configure the NICs for AutoStart to fail them over. You do this by creating additional NIC groups and moving the NICs into the groups where they belong.

AutoStart tracks the state of each NIC in the group. When AutoStart initially detects a NIC, it assumes that the NIC is alive; then the first test takes place.

Configuring a NIC group

You must configure each NIC group’s properties, which include:

◆ The list of NICs that are included in the group.

◆ A subnet address and subnet mask address, which are used for identification purposes.

◆ A set of IP addresses to be used for testing. You must manually configure the NIC group's test IP addresses. All NICs in the group test the same IP addresses. See “NIC testing” on page 39 for more information.

◆ A test frequency.

Any number of NICs can be included in a NIC group, but each NIC can only be a member of one group. If a NIC is not included in a NIC group, it is not tested and cannot be used for NIC-to-NIC failover.

Unplumbed NICs

An unplumbed NIC is not automatically included in a NIC group because it has no assigned base IP address to use for NIC group assignment. However, you can add unplumbed NICs to NIC groups. For optimum performance, add a NIC to a NIC group if it uses the same subnet as the other NICs in the group.

Note: Untested NICs, unplumbed NICs, and NICs on Windows nodes that are included in a NIC group without any configured test IP addresses are represented in the tree view of the AutoStart console with an icon that is different than plumbed NICs and NICs that are tested.

NIC groups 45


NIC group tabs in the AutoStart console

A NIC group’s AutoStart information is provided in the following tabs in the AutoStart console: the NIC Group Details tab and the Testing Options tab. You can see a list of all of an AutoStart domain's NIC groups by viewing the NIC group table.

IP addresses Each resource group needs its own managed IP address, which becomes the virtual IP address that is used for the resource group's application. The managed IP address is a valid, movable IP address. Once a managed IP address is defined, the IP address becomes a managed resource within the AutoStart domain and can be assigned to a node. When a managed IP is assigned, it reacts in the same manner as the node’s static IP address, and applications and other nodes can communicate with the node using the managed IP address.

The details

Normally, each network interface card (NIC) is set to handle one or more base IP addresses. In this way, each machine, or node, maintains its own physical, base IP address. This is the physical IP address that the operating system assigns to a NIC; these addresses are static. These are the IP addresses you use for your servers, and AutoStart uses them to maintain a heartbeat that indicates that the domain is healthy.

AutoStart manages applications that require uninterrupted network communication. Network communication from clients to their application must remain seamless, even when AutoStart relocates an application it is managing to a different node. AutoStart does this by relocating even the IP address to the node. However, it is not the base IP address, but the managed IP address associated with the application that AutoStart relocates. The managed IP is the virtual IP that AutoStart can move. The only movement of a managed IP is:

◆ If a NIC fails, AutoStart moves it to a standby NIC and changes the standby NIC’s usage to Usable. This happens only if NIC-to-NIC is enabled for the failed NIC.

◆ If a resource group becomes unavailable, AutoStart moves it with the application to another node so that the application can communicate with its clients following relocation to another node.

Each of these circumstances is described on the pages that follow.

Managed IP addresses for managed applications

All AutoStart-managed applications that use IP-based communications must use managed IP addresses in AutoStart for communication. You create this managed IP address and add it to the resource group that manages the application. As a result, AutoStart can relocate a managed IP address to the nodes specified for the managed IP address. No matter which node the managed IP address is relocated to, applications and other nodes can communicate with the node using the managed IP address.

You must set up a managed IP address and the nodes it can be assigned to. If a service that normally runs on node A can relocate to run on node B or node C, you must set up a managed IP address for a NIC on each of these three nodes.



Managed IP addresses are standard TCP/IP network addresses with the same behaviors and properties of regular IP addresses, but they are also valid, movable IP address. You define managed IP addresses in AutoStart and assign them to a node; then, AutoStart monitors them, and can relocate them from one node to another if necessary. When AutoStart relocates a managed IP address, the managed IP address behaves in the same manner as the node’s static IP address. Client applications can communicate with the managed IP address because it behaves like a virtual IP address.

Managed IP addresses are important for managing applications that use TCP/IP or UDP IP-based communication. Without managed IP addresses, it is impossible for AutoStart to relocate an IP-based application from one node to another and maintain communication with its networked clients. For an example, see “Example of IP address operation” on page 49.

If there is a NIC failure

You define each managed IP address that will be used in the AutoStart domain. When you define a managed IP address, you specify the target interfaces it can run on; these typically identify the nodes in the valid nodes list for the resource group in which you intend to use the managed IP. For example, if you want AutoStart to be able to relocate the application that runs on Node A to Node B, then you assign a managed IP to target interfaces on Node A and Node B. When you do this, however, you must target on those interfaces whose NIC state is Alive. (If an interface fails over with NIC-to-NIC failover, the managed IP addresses will transfer to the Standby NIC. There are restrictions on Windows nodes, which you can read about in “NIC-to-NIC failover” on page 42.)

AutoStart expects a ping from each IP address in its domain, and waits a number of seconds that you specify to receive that ping.

Use the IP address or name service entry

A managed IP address can be represented to AutoStart in one of two ways, each of which is relocated from node to node during operation. It can be represented in AutoStart as either of the following:

◆ A text-based, name service entry, which is a placeholder name used in the DNS server associated with an IP address. The name service entry can be used and moved from node to node in the same manner as a managed IP address. Instead of defining the IP address as part of the resource group, you specify the name service entry. Then, during operation when AutoStart needs to use the managed IP address, the name service entry is resolved to the managed IP address. Make sure a given name service entry is associated with no more than one IP address in the DNS server. The advantage of having AutoStart use and manage a name service is that it allows the actual IP address to change from a single place.

◆ A numerical IP address represented to AutoStart in the form n.n.n.n. On UNIX nodes (and in environments that need this level of control), AutoStart supports MAC address failover. Defining a managed IP address allows the IP address to be used directly in resource groups and rules, which can then relocate the rule as needed. Once the IP address has been selected, you must immediately register it in the DNS server for hostname resolution.

IP addresses 47


Base IP versus managed IP

Keep in mind that an IP address (base or managed) can be assigned to only one node at a time; it cannot be assigned to more than one node at any one time. For that reason, you must make sure you reserve enough IP addresses in each subnet to cover the subnet's:

◆ Base IP addresses needed in the Windows domain or workgroup

◆ Managed IP addresses that are needed for the AutoStart domains

For example, if the base IP addresses for nodes in the Windows domain or workgroup fall in the range of 1.2.3.1 to 1.2.3.200, the managed IP addresses should fall in the range of 1.2.3.201 to 1.2.3.255. See the “Example of IP address operation” on page 49.

MAC addresses on UNIX nodes

Advanced managed IP address functionality for UNIX nodes specifies a MAC address and subnet mask which is relocated along with the IP address. When this functionality is used, instead of moving the IP address to a virtual interface, the interface on the destination card is plumbed and prepared by assigning the MAC address and subnet mask to the card to act as the original interface. The original configuration of the card is replaced. For more about MAC addresses and UNIX, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Setting the IP limit when you install an agent on a Windows node

On Windows nodes, you must specify the IP limit for each NIC on each node. You can assign up to 255 managed IP addresses to one NIC if the IP limit is configured correctly. (On a UNIX node, you can add addresses to and remove addresses from the NIC without any special configuration.)

When you configure an IP address on a Windows node, you can set the IP limit. Be sure to set the IP limit for the NIC high enough to accept all of the managed IP addresses that may relocate to the node.

By default, for managed IP addresses to work correctly, the address must be on the same subnet as the static IP address of the NIC. For example, if the static IP address of the NIC is set to 1.2.3.4 with a subnet mask of 255.255.255.x, any managed IP addresses that AutoStart uses on that interface card must be in the form 1.2.3.n, where n is any unassigned number from 1 to 255. This functionality is overridden if you configure the IP address to use a specific interface.

Do not change the subnet mask after defining your IP addresses. If you change you subnet, the domain's IP addresses will fail.

If you remove an IP address

An AutoStart agent recognizes only the IP addresses that existed when the agent was installed. If you change a NIC, and you want to use the new IP address as a heartbeat point in a domain line, you must define the new IP address to the node's domain line (on the node's Failure Detection and Mirroring tab). You must remove the old one and add the new one to the domain line in which it is used.



Note that this is true only if, when an IP address becomes unavailable, other domain lines are still available. If all domain lines become available because all have IP address replacements, AutoStart re-detects all the NICs and reestablishes the IP addresses on its own. However, if at least one domain line remains available, AutoStart does not attempt to re-detect NICs and their IP addresses.

IP address tabs in the AutoStart console

An IP address's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab and the Network Path Testing tab. You can see a list of all IP addresses by viewing the IP address table.

Example of IP address operation

Here is an example of how managed IP reassignment works.

Base IP addressesIn this example, a node called iron uses a base IP address of 5.6.7.8. Microsoft SQL server, configured to use the 5.6.7.8 base IP address for its network communication, runs on node iron.

Node iron fails, and AutoStart relocates the resources that are running on iron, including MS SQL Server, to node steel, whose base IP address is 5.6.7.25. Because the application expects the IP address of the SQL Server node to be 5.6.7.8, clients of the movable SQL Server can no longer communicate with it without re-configuration.

Managed IP addressesYou decide to fix this. You configure your SQL Server clients so that they use a managed IP address of 5.6.7.200 for network communication so that SQL Server's clients can now find and communicate with it if AutoStart has to relocate it.

With managed IP addresses, AutoStart can be set up to manage the 5.6.7.200 managed IP address and the application can be configured to use this managed IP address. Now, with SQL Server running and the managed IP address assigned to iron, the node responds to both the base IP address, 5.6.7.8, and the managed IP address 5.6.7.200.

Now iron fails again. The resource group's resources, including the managed IP address and SQL Server, are relocated by AutoStart to node steel. Node steel still responds to its base address 5.6.7.25 but now the managed IP address 5.6.7.200 is also assigned to the node. Because SQL Server clients are configured to use 5.6.7.200 for network communication, clients can locate and communicate with the server without any re-configuration.

For this reason, for all AutoStart-managed applications that use IP-based communications, you must use managed IP addresses in AutoStart for communication.

Node aliases for Windows

Note: Node aliases are available only on Windows.

Node aliases for Windows 49


The node alias function creates an alternate name for Windows-based nodes. Once a node alias is defined and assigned, the node responds to its own NetBIOS name and the node alias name. The node also retains its actual name. Using an alias in a Windows environment is important if an application communicates using the NetBIOS hostname rather than by a specific IP address.

Note: Node aliases are not supported on UNIX systems.

The node alias is a virtual machine or host name for Windows-based application that communicates via NetBIOS protocols, and therefore can be used and moved from node to node. Node aliasing is important if an application communicates using NetBIOS hostname resolution rather than a specific IP address for communication.

For example, you could create a node alias for the Windows browser if you want to browse shares. You can create an alias called FILES to your file server, all the shares under FILES are viewable and accessible. AutoStart monitors the server and if it becomes unavailable you can move the node alias to another server (that is, another node in the resource group) so that your clients still have access to the shares. Obviously, these files would have to be mirrored to another node that the node alias could point to. So you would have to have to keep the files on the healthy node and the passive (that is, standby) node.

Windows will register the service; and it can register it with the server service, the workstation service, or the messenger service. If you register it to the workstation service, you can bind it to a virtual IP address, allowing you to turn it into its own DNS solution.

Once you define a node alias and assign it to its node, the node responds to its own NetBIOS name and the node alias name. The node always retains its actual host name. Using an alias in a Windows environment is important if an application communicates using the NetBIOS hostname rather than by a specific IP address. Node aliases are not available for UNIX systems.

For example, SQL NetBIOS communication is configured using a node’s physical name. Without node aliases, when AutoStart relocates SQL to a new node, communication would stop because the new node has a different name. With node aliases, SQL can be configured to use a node alias name that can relocate along with SQL from node to node.

Once a node alias has been assigned to a node, the alias name appears in the Windows Network Neighborhood listing for the Windows domain as if it were a standard node. A user can select the node as any other real node in the Windows domain. If the node alias is selected in the Network Neighborhood, it displays the same information as the node that node alias is attached to. The node also retains is original name and does not lose that original identity. More than one node alias can be assigned to a node.



Once a node alias is assigned to a node, the alias name appears in the Windows Network Neighborhood as if it were a standard node, as shown in Figure 2.

Figure 2 Node alias in the Windows Network Neighborhood

The node can be selected like any other real node. The node alias may take a few minutes to appear in the Network Neighborhood after it is assigned.

A node alias allows clients that use NetBIOS to look up services when they are moved from one node to another. Rather than configuring the clients to connect to a node's real name, you configure them to access a node alias instead. By doing this, the application services and the node alias move from node to node within the AutoStart domain and the clients can still access it without being reconfigured. Client communication with the application resumes using the alias hostname. However, client machines may have to wait several minutes before re-connecting to shares after relocation. To connect immediately to a share after a relocation, you can clear the NetBIOS cache using the nbtstat -r command. You can clear the IP cache by using the ipconfig /flushdns command.

With node aliases, you can also register the A record or the C name if you need to.

Node aliases for Windows 51


Node alias tabs in the AutoStart console

A node alias’s AutoStart information is provided in the Properties tab in the AutoStart console. You can see a list of all of an AutoStart domain’s node aliases by viewing the node alias table.

Node proxies Node proxies, which are created by default for each node that is added to the AutoStart domain, provide a set of sensors and actuators that monitor and act on a node. A rule can use triggers that monitor a node proxy's sensors. In addition, a rule can use the proxy’s ft_RunProgram actuator to run an application. If an error occurs during execution, a nodeProxy sensor typically returns -1 as its value and an error message as its data. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Node proxy tabs in the AutoStart console

A node proxy’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Options tab, the Sensors tab, and the Actuators tab. You can see a list of all of an AutoStart domain’s node proxies by viewing the node proxy table.


CHAPTER 3State Monitors

This chapter describes response monitors and existence monitors in AutoStart:

◆ State monitors ........................................................................................................ 54◆ Response monitor tests .......................................................................................... 56◆ Existence monitor tests ........................................................................................... 58

State Monitors 53

State Monitors

State monitorsAutoStart provides state monitoring capabilities for processes in the form of existence monitors and response monitors. The existence monitor tests for the physical existence of the process. The response monitor tests for process responsiveness.

It is recommended that the monitor scripts be stored in the AutoStart database so that they can be centrally managed and maintained. However if a monitor script has been distributed across all the nodes, the script can be a single line to execute that local operation.

The state of processes and services defined in the AutoStart domain can be monitored. There are two types of monitors to test different aspects of a process’ health:

◆ Response monitor — The response monitor is an optional test of a process’s response state. The monitor tests the target process to determine if it is healthy and responding correctly. This test is optional, but gives a good indication of a processes’ responsiveness. For more information, see “Response monitor tests” on page 56.

◆ Existence monitor — The existence monitor tests for the physical existence of one or more managed processes. Windows services do not require this type of monitor. User-defined existence monitors for processes provide a more-detailed and accurate status of the process. For more information, see “Existence monitor tests” on page 58.

User-defined versus built-in monitors

A built-in existence monitor checks for the existence of the process ID (PID) number for processes started every second, and uses the Service Control Manager to check for the existence of Windows services. There is also the option of user-defined existence and response monitors for processes and a response monitor for services. Your choices for providing user-defined monitors versus the monitors that are built into AutoStart are listed below.

Resource Monitor Built-in User-defined None

Services Response x x

Existence x

Processes Response x x

Existence x x

Process proxies Response x x

Existence x x

Node proxies Response x

Existence x

Data sources Response xa x

Existence x

a. Not all data sources have built-in response monitors.


State Monitors

These user-defined monitors can provide a highly-detailed and accurate status of the process or service. The monitor tests may be written as a Perl script, or a Bourne Shell script, or a Windows batch command script. Once saved, the scripts are stored in the AutoStart database, allowing access from all nodes. For more about user-defined state monitors, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Each managed process or service is constantly monitored by AutoStart. AutoStart returns a status to the AutoStart console and, based on the configuration, takes action if the process or service fails. Understanding the possible reported statuses and how the information is reported to the AutoStart console helps you to understand where to look for problems.

A more detailed look at state monitors

The existence and response tests provide the information that tie directly to service and process statuses:

◆ The existence test asks: Do ANY of these processes or services exist?

◆ The response test asks: Do ALL of these processes or services exist and are they functioning properly?

If the answer to a test is Yes, the test is true and returns a running status. Obviously, the response test is usually more involved than just checking for all of the required processes, but a simple test can check simply for the existence of all child and parent processes needed for the application to run. If the process is in the Running, No Response, or Stopping state, you cannot start it somewhere else.

By default, there is no user-defined existence monitor, so AutoStart checks for the existence of the Process ID (PID) number AutoStart documented when it started the process, every one second. For services it uses the Service Control Manager. In addition to this default, you can use the following user-defined tests to provide a more-detailed and accurate process or service status:

◆ User-defined existence and response monitors for processes. If the process creates a child process or contains multiple processes, you must create an existence monitor. The default monitor does not detect these additional processes automatically.

◆ A user-defined response monitor for services.

Writing monitor tests

You can write monitor tests using a Perl script, Bourne Shell script, or a Windows batch command script. AutoStart stores saved scripts in the AutoStart database, allowing all nodes to access scripts without manually copying files to each server. A basic understanding of what existence and response tests to look for and when you need them helps you avoid most common problems:

◆ The default existence test monitors only the PID of the process spawned by AutoStart. An application that contains multiple process executables needs a valid existence monitor associated with it to watch the child processes, or it is possible for the top level process to exit and cause AutoStart to report a failed state even when the user still sees the application running. A common symptom of this is when the user starts a process and it transitions to a Running state temporarily before transitioning to Failed.

State monitors 55

State Monitors

◆ Confusing the purpose of the two types of monitors can result in false failure reports and multiple instances of the application running on different nodes. There is only one purpose for an Existence test; to determine if ANY part of the managed application still exists on the machine. For example, if an application has three processes, the test returns success even if only one of the processes is still running and the application is totally unresponsive. To better understand this issue, think about what happens when AutoStart issues the application stop command to the application. The process receives the command and transitions to the stopping state where it stays until the existence test reports that everything is gone. If the existence test is not correct, AutoStart reports the process as stopped before it is really gone, which can lead to problems.

◆ The Response Monitor is responsible for testing the health and responsiveness of the application and typically attempts to connect to it. You should use the response test to ensure that all the processes comprising a multi-process application are running. If they are all running, use the response test to connect to the application and determine its health. When a response test reports a failure, AutoStart reports a process state of No Response. If configured to do so, AutoStart stops and restarts the application when it returns this state.

◆ When the existence or response script type is Shell (instead of Perl), AutoStart uses the default shell on the system. It runs whichever shell is associated with the program sh. To ensure that the correct shell is running, specify the shell type using the #!<path> statement as the first line of the shell script.

◆ When AutoStart runs scripts and programs under a particular user account, it only sets the user ID information on the child process. It does not load the user’s default environment in either NT or UNIX. As a result, you should specify all necessary environment variables in the root user’s default environment to ensure that the AutoStart agent reads and passes them down to any child processes. You can also specify them as part of the process configuration. A test must return the value of zero for passing and a value of one for failure.

State monitor tabs in the AutoStart console

A state monitor's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab and the Test Script tab. You can see a list of all of an AutoStart domain's services by viewing the state monitor table.

Response monitor testsThe response monitor checks the health of a process, to ensure that the process is running as it should and is not in a hung or otherwise unresponsive state. The response test determines an application's health by verifying communication and functionality.


State Monitors

When a response monitor reports a failure

When the response test succeeds, it continues monitoring the application’s health; but if it fails, it changes the state of the resource it is testing. Here is what happens to the state of a process:

After the process is set to the No Response state, the response test continues to run. If, at any time, the response test runs to completion without a timeout, and the test passes, the state of the process is transitioned back to Running. If the response monitor times out two consecutive times, the process transitions to the No Response state.

You can configure AutoStart to stop and restart the process when it transitions to No Response. To do this, use the process configuration’s Options tab in the AutoStart console and click the tab’s Help button for more information.

Built-in versus custom response monitors

AutoStart provides a built-in response monitors for the resources listed below. However, you can provide customized response monitors for services and processes, or you can choose to use no response monitoring at all in some cases, as described here:

Guidelines for custom response monitors

The following guidelines are specific to custom response monitors for services, processes, and process proxies.

The response monitor for each process must be written specifically for that process. Because the activities that define whether a process is running normally depends on the type of process being run, there is no single test that could be used for all processes. For example, the response monitor for a database process might be written to query a table in the database, while the response monitor for a web server may send an HTTP request to the server and wait for a response. In each case, the test should be enough to ensure that the process is working correctly, but not create too much overhead.

Original state State if response monitor succeeds

State if response monitor fails

Running Running No Response

Stopping Stopping Stopping

No Response Running No Response

Built-in Custom None

Services x x

Processes x x

Process proxies x x

Node proxies x

Data sources xa x

a. Not all data sources have built-in response monitors. For more information, refer to the documentation for the data source you are using.

Response monitor tests 57

State Monitors

When creating the response monitor, check the response of each individual process which makes up an application. If the application is made up of multiple processes, the monitor should be written so that it fails if any of the processes are unresponsive.

Existence monitor testsThe existence monitor is intended to check if the process or processes that make up an application are still running on the node, and if any part of the managed application still exists on a node.

Note: Do not use an existence test to try to connect to the application and test its health or responsiveness, because that's what a response monitor does. For more about testing for application health, see “Response monitor tests” on page 56.

The interval at which the monitor test runs is determined by the Test Run Interval, which you can set by using the monitor’s Settings tab in the AutoStart console. Typically, the existence monitor interval is smaller than the response monitor interval because the existence monitor checks actual existence of the process, ensuring that the process is online. For existence monitors, the test run interval is equal to the maximum amount of time it takes to transition the process to the Stopped state.

AutoStart provides a default existence monitor that can be used with simple processes, however its scope is limited. With a custom existence monitor, applications with multiple processes can be tracked, or multiple processes can be launched by a startup script.

Built-in versus custom existence monitors

AutoStart provides a built-in existence monitor for the resources listed below. However, the built-in existence monitor may not be adequate for the processes you are monitoring, in which case you must provide a custom existence monitor that you or EMC Customer Support has provided.

The built-in existence monitor

AutoStart provides a default existence monitor for every process. This test checks for the existence of the process ID (PID) number assigned to the process that is started by AutoStart. The built-in existence test doesn't monitor any process other than the initially-started process—it doesn't monitor any child processes. For this reason, the default existence test is not be adequate for use with all processes, which means that you may need to create a custom existence monitor.

Built-in Custom

Services x

Processes x x

Process proxies x x

Node proxies x

Data sources x


State Monitors

Custom existence monitors for processes and process proxies

The default existence monitor cannot be used in the following circumstances:

◆ to monitor multiple processes started by a startup script, or

◆ for a managed application that is made up of multiple processes, or

◆ when the process that is started is not the process to be monitored.

For example, Oracle uses processes that are integral to its correct operation, and for that reason an existence monitor is required with this application to ensure that all of the necessary processes exist.

Here is why a custom existence test is necessary: The built-in existence text script runs, starting up the other processes, and then exits. Only the PID of the startup script is known; there is no record of the PID numbers for subsequent processes. When the script exits after starting the processes, AutoStart interprets the exit of the script as a failure and runs the script again. Furthermore, when the process starts the other (child) processes, the default existence monitor cannot monitor the child PID numbers. For that reason, if any of the child processes fails, the failure will not be recognized. To avoid this undesirable behavior, you must create and use a custom existence monitor.

Guidelines for custom existence monitors

The following guidelines pertain specifically to existence monitors. The existence monitor can be more general than the response test for a process. Generally, if a monitor detects a process, the process is considered to exist. Typically, the existence of a PID or the process name on the node is enough to determine existence. AutoStart supplies tools to assist in determining if the process exists by searching for process name or PID.

Note: If the existence monitor test fails, the process transitions to the Failed state. Once the process is in this state, the stop command defined for the process cannot be run, because the process is considered failed. Therefore, when you write an existence monitor test, ensure that the existence test takes the proper action before the process is declared Failed.

If the application is comprised of a number of processes, the existence test should not fail unless all of the instances of the processes are no longer in existence. If any trace of the process can be found, the process still technically exists. For example, if you are checking for the existence of an application called MyApp which uses two processes, my1 and my2, a sample test written in Perl may look like the following:

require "$ENV{FT_DIR}/bin/monitor_tools.pl";if (($ft::ProcExists{"my1"}) || ($ft::ProcExists{"my2"})) {# Success!exit(0);}else {# Failure!exit(1)}

The value of $ft::ProcExists is the PID number, a positive integer if the process exists, and the expression evaluates true if a PID is returned. If either process exists, the test passes.

Existence monitor tests 59

State Monitors

Processes in the Running or No Response state transition to the Failed state when the existence monitor fails, as shown here:

Existence monitor timeout

If the existence monitor times out, the state does not change. However, you must be careful to write the existence monitor in such a way that it does not time out. The test must run to completion before any determination of the process’ existence can be made. A test that keeps timing out prevents that determination from ever taking place. When you test existence monitor tests, keep an eye on the event log to make sure that the monitor is not timing out.

Original state State if existence monitor succeeds

State if existence monitor fails

Running Running Failed

Stopping Stopping Stopped

No Response No Response Failed


CHAPTER 4Data Sources

This chapter describes how data sources are managed in AutoStart, and describes the types of data sources AutoStart supports:

◆ Data sources........................................................................................................... 62◆ The anatomy of an attach ........................................................................................ 63◆ Parallel attach of data sources ................................................................................ 64◆ AutoStart data sources for RepliStor 6.x .................................................................. 67◆ AutoStart data sources for RepliStor 5.x .................................................................. 68◆ Composite data sources.......................................................................................... 69◆ EMC Mirroring for Windows data source .................................................................. 71◆ EMC MirrorView mirroring data source..................................................................... 73◆ EMC SRDF Mirroring data source.............................................................................. 76◆ Mount points data source for Windows ................................................................... 85◆ Shared Disk Device for Windows data sources......................................................... 86◆ VERITAS Volume Manager for Windows data sources ............................................... 87◆ VERITAS Volume Manager for Solaris data sources................................................... 87◆ Windows Network Share data sources..................................................................... 88◆ AIX Volume Group data sources............................................................................... 89◆ HP-UX LVM data sources.......................................................................................... 90◆ NFS data sources .................................................................................................... 91◆ Sun Solstice DiskSuite data source ......................................................................... 91◆ UNIX file system type data sources.......................................................................... 92

Data Sources 61

Data Sources

Data sourcesIn many distributed configurations, data storage is on a common storage device separate from the node. You define data sources to AutoStart that identify these common storage devices so that AutoStart can control access to the data; as a result, in the event of failure, a node is not cut off from the data. The AutoStart data source provides data content to an application that AutoStart manages.

By defining a data source, you identify the object through which AutoStart can monitor and manage a disk resource. The name you give to the data source is the common name AutoStart uses to refer to the data source. Using this common data source name, AutoStart servers can attach and detach the disk resource, as well as query and monitor its state. Once the data source is attached to an AutoStart server, that server can read and write data to the storage device by way of the data source definition.

Data access

AutoStart manages access to the storage device, making sure that concurrent access does not corrupt data. However, AutoStart does not control disk operations or provide any access restrictions. Permissions and restrictions are controlled by the underlying operating system or file system. Attach and detach operations are done through standard operating system (OS) commands (on UNIX) and APIs (on Windows). AutoStart does not in any way insert itself into the data path of the managed application; once the data source is attached to a node in the AutoStart domain, that node can then read and write data to the storage device. AutoStart does not provide a clustered file system.

Attaching

When attempting to attach to a specified data source, a file system-specific test is run in order to verify that the drive is working and available. This and the data source's existence and response tests are discussed in detail in “The anatomy of an attach” on page 63.

Note: In some cases, a resource group may contain multiple data sources. It is important that you sequence the data sources correctly in the startup and shutdown sequences.

List of data sources

AutoStart supports a wide range of data sources for both Windows and UNIX, including: file systems, volume managers, and data. See Table 1 for the data sources that are supported on the platform you are using.

Table 1 Data sources and their operating systems

Data sources Windows Solaris Linux AIX HP-UX

AIX Volume Group x

Composite Data Source x

Data Source for RepliStor® 5.x x

Data Source for RepliStor 6.x x

EMC Mirroring for Windows x


Data Sources

Each data source is described in this chapter.

The anatomy of an attachWhenever you or AutoStart attempts to attach a data source, the first step of the attach is a file-system test, which runs to verify that the drive where the data source will be attaching is working and available. The commands that the attach command runs differ depending on the operating system where the data source is attaching:

◆ On UNIX file systems, the fsck command runs. An initial fsck command runs to determine if a full fsck needs to be performed on the disk. If the full test is needed, a full fsck is performed on the drive. Once the fsck has completed successfully, the data source state is attached. If the fsck fails, the data source is not attached, and an error is logged in the event log.

◆ On Windows disks, the CHKDSK facility runs. (For Windows shared disks and mirrors, the facility can be disabled.) AutoStart begins by enabling the device, and then performing a CHKNTFS command to see if a full CHKDSK command needs to be run. If CHKDSK fails, the data source does not attach, and an error is logged in the event log. On FAT drives (non-NTFS drives), a CHKDSK is run by default without determining the state of the drive. It is possible to set the parameters of the CHKDSK command.

This determines that the drive works.

The response and existence tests

All data sources have a continual existence test that determines that the file system or the volume has not been unmounted. AutoStart continually uses the built-in existence test to make sure the file system or volume is mounted.

EMC MirrorView™ Mirroring x x

EMC SRDF Mirroring x x x x x

Data sources Windows Solaris Linux AIX HP-UX

HP-UX LVM x

Mount Points x

NFS x x x x

Shared Disk Device for Windows x

Sun Solstice DiskSuite x

UNIX File Systems UFS, VxFS ext2, ext3, hpfs, reiser

JFS HFS, VxFS

VERITAS Volume Manager for Solaris

x

VERITAS Volume Manager for Windows

x

Windows Network Share x

Table 1 Data sources and their operating systems (continued)

The anatomy of an attach 63

Data Sources

Then there is the response test. All data sources include a built-in response test that verifies that the node can properly read or write to the file system or the volume. For some data sources, this response test is optional. After a data source attaches, the response test runs at an interval specified in the data source configuration.

Each time the response test runs, a file is written to the disk and data is read from the disk. If the test runs successfully, no state change occurs. If, after two consecutive attempts, either portion of the test does not complete successfully, the state transitions to No Response.

Windows Network Share, UNIX file system, or Shared Disk Device for Windows data source types

When the response test runs for the first time on a Windows Network Share, UNIX file system, or Shared Disk Device for Windows data source types, it creates a directory on the file system that AutoStart uses for the file it creates for the test. You specify the interval at which the test is run. Each time the test runs, it writes a file to the disk and reads data from the disk. If the test runs successfully, no state changes occur. If, after two consecutive attempts, either portion of the test does not complete successfully, the state transitions to No Response.

At the end of the response test, AutoStart deletes the test file. The directory the test creates remains on the file system at all times.

AutoStart uses the response test by default, but you may need to turn it off. Why? Because the response test uses Windows-, UNIX-, or Linux-standard APIs to go out and write and read from the drive. You may have a new disk drive protocol that works fine with the operating system but the AutoStart API is not able to talk specifically to the drive protocol because the protocol is new. In this case, the response test can fail. However, if everything worked prior to starting AutoStart (and the assumption is that everything did work), you can turn off the response test.

Use the built-in response test for standard UNIX and Windows data source implementations. If you are using propriety implementation for your application, the built-in response test may fail. If it fails, turn it off. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: Although you can create user-defined response tests for use with services, processes, and process proxies, you must use the built-in response test for data sources, or no response test at all.

Parallel attach of data sourcesWhen multiple data sources are included in a resource group, they can attach simultaneously, using the parallel attach feature. This feature is useful if you have more than one independent data source and you would like to improve the efficiency of bringing the resource group online by having more than one data source attach operation occur in parallel.

Selecting parallel attach causes the resource group to begin the attach operation and immediately move on to the next step in the startup sequence without waiting for the data source to transition to the attached state. The advantage is that you can reduce the time it


Data Sources

takes to attach these data sources by having the attaches occur at the same time. If each of the file systems requires a lengthy CHKDSK, this would significantly reduce the time it takes for the resource group to come online. You select the parallel attach option when you add a data source to a resource group.

Do not use parallel attach for a Replistor data source. Doing so may cause the data source to fail when the network is busy.

Used with a data source checkpoint

A data source checkpoint provides a point in the startup sequence that causes the resource group to pause until all of the data sources that precede it have transitioned to the Attached state. When building the startup sequence, you should add a data source checkpoint to the sequence immediately prior to any resource group object that requires access to the file system data sources. This gives the network file systems time to transition to the Attached state before moving forward.

You may need to include multiple data source checkpoints in the startup sequence, depending on your environment. For example, assume that you have three volume groups with three file systems on those volume groups. Rather than waiting for each volume group to attach, then waiting for each file system to attach, you could arrange the startup sequence to:

1. Attach all of the volume groups in parallel.

2. Wait for them to transition to attached.

3. Attach all of the file systems in parallel.

The advantage is that you can reduce the time it takes to attach these data sources by having the attaches occur at the same time. If each of the file systems requires a lengthy CHKDSK, this would significantly reduce the time it takes for the resource group to come online.

The Parallel Attach feature, used in conjunction with the Data Source Checkpoint feature accomplishes this:

1. Select parallel attach for each independent volume group data source.

2. Then, before the file system data sources, add a data source checkpoint so that the file systems are not mounted before the volume group is ready for them.

3. Then add another data source checkpoint before any resource that requires the file systems to be attached to access their data.

For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Example

For example, assume that you have a resource group that has three network file system data sources. By setting the parallel attach feature of each data source, each of the network operations can proceed concurrently. You can add a data source checkpoint before a resource that requires access to the file systems; this gives the network file

Parallel attach of data sources 65

Data Sources

systems time to transition to the Attached state before moving forward. The data source checkpoint is a point in the startup sequence at which the resource group will pause until all of the data sources that precede it have attached.

You may choose to have multiple data source checkpoints depending on your environment. For example, assume that you have three volume groups and three file systems on those volume groups. Rather than wait for each volume group to attach, then waiting for each file system to attach, you could arrange the startup sequence to attach all of the volume groups in parallel, wait for them to transition to attached, then attach all of the file systems in parallel.

Take the following volume group (VG) and file system (FS) data sources:

In this case, the most efficient way of defining the resource group is to put resources in the following startup sequence:

1. VG1 data source with parallel attach (relies on no other resource)



4. Data source checkpoint

5. FS1 data source with parallel attach (relies on VG1 being attached)



8. Data source checkpoint

9. SRV1 service (relies on FS1 being attached)



Data source Device

VG1 dev1

VG2 dev2

VG3 dev3

Data source Device Volume group

FS1 dev1 VG1

FS2 dev2 VG2

FS3 dev3 VG3

Service File system needed

SRV1 FS1

SRV2 FS2

SRV3 FS3


Data Sources

In this example:

1. First the three volume groups (VG) attach concurrently (because of the parallel attach option), offering up to three times the throughput.

2. Then the first data source checkpoint is reached; the resource group waits for all of the data sources that precede it to transition to the Attached state. This data source checkpoint allows the volume groups to attach before the subsequent file systems, which reside on the volume groups, attach; it is critical that the volume groups be attached before the file systems attach.

3. Once all three volume groups are attached, the three file systems (FS) attach concurrently (because of the parallel attach option), again offering up to three times the throughput. Because the volume groups are already attached, these file systems, which reside on them, are successfully mounted.

4. The second data source checkpoint is reached; the resource group waits for all file systems to attach successfully.

5. Finally, the services, which rely on the file systems being attached, go online.

AutoStart data sources for RepliStor 6.xThe AutoStart Data Source for RepliStor 6.x is a Windows-only data source you can use with EMC RepliStor 6.x. RepliStor is an EMC replication product that replicates at the file level. RepliStor replicates files, directories, shares, and registry keys from a source system (the computer that has the data that needs to be protected), to a target system (the computer to which the data is replicated). After an initial synchronization of the data, RepliStor replicates any changes made to the files (or keys), transfers those changes to the target system, and updates the files on the target system.

Note: If RepliStor 5.x is installed, you cannot use this data source. Use the “AutoStart data sources for RepliStor 5.x” on page 68 instead.

You can set up AutoStart to manage and monitor RepliStor replication. You can:

◆ Create a data source for RepliStor 6.x that lets AutoStart manage a RepliStor specification.

◆ Create and delete RepliStor specifications from within the AutoStart console.

◆ You can manage a RepliStor data source as a stand-alone data source. Or you can add it to a resource group or module instance, to use the source files that have been mirrored to the target so, if there is a failover from the source to the target, you have identical data.

◆ Include a data source for RepliStor 6.x in a Composite data source. For more information, see “Composite data sources” on page 69.

Working with RepliStor 6.x data sources in AutoStart

When you work with a RepliStor 6.x data source in AutoStart:

◆ Before you do anything with RepliStor 6.x data sources in AutoStart, you must set up RepliStor login credentials for each RepliStor site that AutoStart needs to access.

AutoStart data sources for RepliStor 6.x 67

Data Sources

◆ Before you can begin replicating, you must create a specification that tells RepliStor software which files, directories, Registry keys, and shares to replicate. For how to create specifications, see AutoStart online help and the EMC AutoStart Administrator’s Guide. At any time, you can see the RepliStor specifications that have been defined.

◆ Once you have defined the specification, you can create the data source in AutoStart.

◆ When your data source is complete, you can attach it, which lets you start replicating. You can also include it in a Composite data source or in a resource group.

◆ If you use RepliStor to delete a specification that AutoStart is using, you'll find that you can't work with the AutoStart data source anymore. You'll need to re-create the specification by following the steps in AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ Other possible stumbling blocks and their solutions are provided in Troubleshooting discussions in AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ If you need to remove a RepliStor specification from AutoStart management, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide.

RepliStor server failure

Once the RepliStor data source is attached, it starts replicating. If the RepliStor server fails on either the local or remote node, the RepliStor 6.x data source enters the Warning state (yellow state icon). The Warning state indicates that there is a problem with the RepliStor server that must be corrected; the RepliStor 6.x data source, if in a resource group, will not fail over. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

AutoStart data source for RepliStor 6.x data source tabs in the AutoStart console

An AutoStart Data Source for RepliStor 6.x data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, the Spec Management tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

AutoStart data sources for RepliStor 5.xThe AutoStart Data Source for RepliStor 5.x is a Windows-only data source you can use with EMC RepliStor 5.x. RepliStor 5.x must be installed on both servers. Before you can create a data source for RepliStor 5.x, you must make sure that RepliStor 5.x is set up as described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: If RepliStor 6.x is installed, you cannot use this data source. Use the “AutoStart data sources for RepliStor 6.x” on page 67 instead.


Data Sources

AutoStart Data Source for RepliStor 5.x Data Source tabs in the AutoStart console

An AutoStart Data Source for RepliStor 5.x data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

Composite data sourcesYou may have a situation in which you have local synchronous data sources on two or more nodes that, if they fail, you want restarted on a remote node where asynchronous storage is kept. You can manage such a situation by creating a Composite data source that you use in the resource group that manages the nodes. The Composite data source, which is available on Windows only, is an abstract data source that manages “real” data sources by combining individual data sources into a single data source that can work with three or more nodes.

You can use a Composite data source to combine the following types of Windows-only data sources:

◆ “Shared Disk Device for Windows data sources” on page 86

◆ “EMC Mirroring for Windows data source” on page 71

◆ “VERITAS Volume Manager for Windows data sources” on page 87

◆ “AutoStart data sources for RepliStor 6.x” on page 67

Note: Although an SRDF device is a shared disk device, it cannot be used in a Composite data source.

Failover

How the Composite data source fails over is illustrated in “Example of failover using a Composite data source” on page 70. If you are using a RepliStor 6.x to replicate to your remote failover node, when one of your mirroring nodes (A or B) does become available (as described in the example), you must manually initiate recovery. How you do this is described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Critical Composite data sources

A Composite data source goes to a Warning state when one of its data sources fails, unless you have defined the data source that failed as critical. When a critical data sources fails, the Composite data source goes to the Failed state. A Composite data source that enters the Failed state causes the resource group to restart on another node. In the example “Example of failover using a Composite data source” on page 70, you would define the EMC Mirroring data source (AB) as a critical data source because you would want failover to occur if AB failed. But the RepliStor data sources are not critical data sources; if they fail, the resource group will go into a Warning state, but will not failover.

You identify a Composite data source’s critical data sources by using an environment variable whose value you set to the name of the data source that is critical. Any number of data sources can be defined as critical.

Composite data sources 69

Data Sources

The steps for configuring a Composite data source and identifying its critical data sources are described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Environment variables for Composite data sources

For Composite data sources, see the following topics in AutoStart online Help to learn how you can use environment variables to:

◆ “Identify the Composite's critical data sources”

◆ “Environment variables for Composite data sources”

Composite data source tabs in the AutoStart console

A Composite data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

Example of failover using a Composite data source

You have an EMC Mirroring for Windows data source with a source on node A and a target on node B, the local failover node which is where your applications are restarted if node A becomes unavailable. RepliStor provides asynchronous replication from node A to node C. If nodes A and B become unavailable, you want your applications restarted on node C, using the RepliStor data source on node C, the remote failover node:

◆ An EMC Mirroring data source (called AB) manages synchronous replication from node A to node B.

◆ A RepliStor data source (called AC) manages asynchronous replication to data storage from node A to node C.

◆ Another RepliStor data source (called BC) manages asynchronous replication to data storage from node B to node C.

If node A becomes unavailable, the resource group is restarted on node B. But if there is a total local failure (both nodes A and B become unavailable), the applications will restart on node C. Failover will occur in this sequence, and failback will also occur.

So that this behavior will occur, you add all three data sources (AB, AC, and BC) to a Composite data source called ABACBC, then include the Composite data source in the resource group that manages the application. Then you attach the Composite data source to the node where you want the data source to come online.


Data Sources

Three examples are depicted in Figure 1; the node that ABACBC is attached to is shaded.

Figure 1 Composite data source example

The examples are:

◆ Normal operation — For normal operation you attach ABACBC (Composite) to node A. In this case, data sources AB (EMC Mirroring) and AC (RepliStor) are enabled.

◆ Node A becomes unavailable — If node A becomes unavailable, ABACBC (Composite) is attached to local node B where the resource group restarts when it fails over. In this case, data sources AB (EMC Mirroring) and BC (RepliStor) are enabled.

◆ Both A and B become unavailable — But if both nodes A and B become unavailable, ABACBC (Composite) is attached to remote node C, where the resource group restarts after a total local failure of nodes A and B. RepliStor data sources AC and BC are enabled.

In this case, though, node A and node B are not available, so no replication occurs. However, the remote failover server on node C will keep track of all changes.

Recovery from a remote failover node

If, as in the example, you are using a RepliStor 6.x to replicate to your remote failover node, when one of your mirroring nodes (A or B) does become available as described in the example above, you must manually initiate recovery. How you do this is described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

EMC Mirroring for Windows data sourceThe EMC Mirroring for Windows data source is the vehicle you use for attaching your EMC Mirroring for Windows mirrored nodes. Before you can create a data source for EMC Mirroring for Windows, you must make sure the mirror (that is, the source and target nodes that comprise the mirror) is configured correctly. For example, if you are using the data

EMC Mirroring for Windows data source 71

Data Sources

source in a resource group created using AutoStart's Windows Print Services module, refer to additional information in “Windows Print Services module” on page 116.

Once you create an EMC Mirroring for Windows data source, you can attach it. Once attached, the EMC Mirroring for Windows data source begins mirroring. For a complete list of steps, refer to AutoStart online help or the EMC AutoStart Administrator’s Guide.

Keep in mind that shares that have been defined prior to defining the EMC Mirroring for Windows data source are captured only after the data source is attached to the source node. Once the data source is attached to the source node, the AutoStart agent monitors for new and deleted shares. If AutoStart relocates the data source, permissions relocate with the data source, along with the shares.

Note: You can include a EMC Mirroring for Windows data source in a “Composite data sources” on page 69.

EMC Mirroring for Windows

AutoStart provides the EMC Mirroring for Windows data source, a block-level (volume-level) mirroring device for Windows servers, built into AutoStart. The purpose of the EMC Mirroring for Windows data source is to replicate data from a source drive to a target drive. No separate license is required to use this data source. Mirroring is synchronous and is only between two nodes. Furthermore, you are limited in that you can have only 1 mirror line on each of the 2 AutoStart nodes. Thus, all mirroring for this data source can only be between the same 2 nodes; no 3-way mirroring is available (nor is three or more-way mirroring available).

Note: Do not use the EMC Mirroring feature over a wide-area network (WAN). To mirror over a WAN, you can use EMC SRDF Mirroring or RepliStor, which are replicating data sources that AutoStart supports. For more information, see “EMC SRDF Mirroring data source” on page 76 or “AutoStart data sources for RepliStor 6.x” on page 67.

Be aware that EMC Mirroring for Windows provides very basic mirroring. Each partition is segmented into blocks, which are replicated. These blocks contain all of the bit information for any data on the partition.

Initial mirroring activity

Mirroring begins after you create the EMC Mirroring for Windows data source and attach it (which is described later in this section). Initially, the mirroring makes an exact copy of each data block in a source's partition. It then writes these copies to a given target partition on the second server within a cluster. It takes each and every block on the source disk and writes it over each and every block on the target disk. The target disk is completely overwritten.

In this way, it mirrors a source disk to its target disk; for example, it mirrors node A to node B. During this mirroring process, it maintains synchronization; any file write that occurs on the source is mirrored to the target.

Note: Permissions migrate from the source to the target.


Data Sources

Subsequent changes

When a data change occurs to an area that exists on the partition, AutoStart captures and then updates the effected block or series of blocks within the partition. Any change that occurs on the source partition is applied to blocks on the target partition. Because all information is stored on these data blocks within a given partition, all information is replicated and updated as long as the servers continually communicate over the dedicated link, and as long as the data source remains attached.

You can also set up AutoStart to warn you when a disk is getting full.

EMC Mirroring for Windows data source tabs in the AutoStart console

An EMC Mirroring for Windows data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

EMC MirrorView mirroring data sourceThe EMC MirrorView Mirroring data source is a software module that is installed with EMC AutoStart, which you can enable for use with EMC CLARiiON® MirrorView/Synchronous (MirrorView/S) or MirrorView/Asynchronous (MirrorView/A). The EMC MirrorView Mirroring data source manages CLARiiON MirrorView logical unit numbers (LUNs) in AutoStart environments, where copied images of the LUNs are kept at separate locations in the event of disaster recovery. When a host (an AutoStart node with an installed agent added to the AutoStart configuration) or a CLARiiON failure occurs, MirrorView assists in the recovery effort.

Note: You can use the EMC MirrorView Mirroring data source only if it is enabled with a valid license key.

MirrorView/S

MirrorView/S (MirrorView/Synchronous) is a disaster-recovery solution that continuously mirrors data in real-time from a local site to a remote disaster recovery site. MirrorView/S is ideal when recovering data for critical applications over shorter distances. As the host writes data on the primary site, the CLARiiON concurrently writes the data to the secondary or remote site. If a disaster occurs, the data on the secondary site exactly matches the data on the primary site. For MirrorView/S the links between primary and secondary sites must handle peak workload bandwidths. MirrorView/S ensures availability of an exact up-to-date copy of data.

Figure 2 shows a MirrorView/S flow, which goes like this:

1. Write data from the server (Exchange) transfers to the primary image (host A).

2. Data writes to the primary image (host A) and the secondary image (host B) simultaneously.

3. The secondary image (host B) acknowledges receipt back to the primary image (host A).

EMC MirrorView mirroring data source 73

Data Sources

4. An acknowledgement is sent to the server.

Figure 2 MirrorView/Synchronous example

MirrorView/A

Designed for low-bandwidth requirements, MirrorView/A (MirrorView/Asynchronous) provides highly available storage across the globe without distance limitation. MirrorView/A periodically updates a remote copy of production data by writing data from a local site to a remote CLARiiON server in the background. You can use the resulting mirror for remote data bunkering and simplified, remote disaster recovery.

You must complete the checklist

Before you can set up a MirrorView data source, you must complete the checklist, which you can find in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Basic MirrorView information

MirrorView provides disaster recovery through the maintenance and mirroring of LUNs (images) on other storage systems. In the MirrorView environment, a mirror refers to multiple mirror images of the primary LUN. There is a primary image and zero or more secondary images. Each image operates in a separate storage system, thus the storage system for the primary image is referred to as the primary storage system, or local storage system. The secondary image is referred to as the secondary storage system or the remote storage system.

There is one MirrorView image per system. For example, on CLARiiON 1 you would have a primary image which is unique to the secondary image on CLARiiON 2. This primary image and secondary image form one unique image. You could have additional secondary images, however these would be on additional CLARiiONs.

The AutoStart MirrorView data source manages CLARiiON MirrorView systems in an AutoStart environment. MirrorView operates bi-directionally over a storage area network (SAN). Each CLARiiON system consists of two storage processors: SPA and SPB, identified by an IP address or hostname. The AutoStart “Shared Disk Device for Windows data sources” on page 86 or “VERITAS Volume Manager for Windows data sources” on page 87 manages access to the CLARiiON devices on Windows systems of more than two nodes; the AutoStart “VERITAS Volume Manager for Solaris data sources” on page 87 manages access on Solaris systems of more than two nodes.


Data Sources

Note: On Linux, configurations of more than two nodes require that you devise a solution of your own design; contact your EMC representative for more information.

The term LUN refers to the primary image (other names are local image or production image) on a storage system, and the secondary image (other names are remote image or copy image) on a second storage system(s). MirrorView supports up to two remote images at a time, but operates on one at a time. Image states include the following:

◆ In-Sync state — Data on the secondary image is identical to data on the primary image; with a new write the state changes from In-Sync to Consistent.

◆ Consistent state — The secondary image is identical to the primary image up until a point in time, which is typically 30-60 seconds.

◆ Out-of-Sync state — The secondary image differs from the primary image. Full synchronization is required to make the secondary image usable for recovery operations.

◆ Synchronizing state — The process of updating a secondary image with primary image data. Until synchronization completes, an image cannot be promoted to primary (to promote a mirror means to ensure that data fails over properly).

When communication fails between the primary and secondary images, the secondary image becomes fractured. In turn, MirrorView records the failed write to the primary LUN in a Fracture Log, which is a bitmap that indicates which portions of the primary image differ from the secondary image(s). As long as the secondary LUN remains unreachable, a separate fracture log tracks changes for each region in the primary LUN (when the primary image loses contact with the secondary image, the secondary image becomes system fractured). When each secondary image becomes available, recovery time is minimized. After synchronization completes, data is identical on the primary and secondary LUNs.

Before you create an instance of an EMC MirrorView Mirroring data source, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for a checklist of prerequisites, requirements, and a pre-configuration checklist.

EMC MirrorView data source tabs in the AutoStart console

An EMC MirrorView mirroring data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Administration tab, the MirrorView Settings tab, the Advanced tab, and the Status tab. There are right-click menu options available in the directory tree, as described in Data source Action menu and toolbar. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

MirrorView data source features and terminology

The MirrorView data source provides the following features:

◆ Support for synchronous consistency groups.

EMC MirrorView mirroring data source 75

Data Sources

◆ Parallel attach support: When multiple data sources are included in a resource group, you can choose to have multiple data source attaches occurring simultaneously by using the Parallel Attach feature. Selecting this checkbox causes the resource group to begin the attach operation and immediately move on to the next step in the startup sequence without waiting for the data source to transition to the attached state.

◆ This feature is useful if you have more than one independent data source and you would like to improve the efficiency of bringing the resource group online by having more than one data source attach operation occur in parallel.

◆ Supports MirrorView/Synchronous and MirrorView/Asynchronous replication, when connected to a pair of CLARiiON storage systems.

◆ Allows AutoStart to manage local failover and remote failover of critical applications.

◆ Allows AutoStart to manage MirrorView failback operations.

◆ Supports all AutoStart modules (active/active and active/passive). An active/active configuration means that both sites contain production data, and each site is a disaster recovery site for its peer. Active/passive denotes a production environment for one site. Disaster recovery is at another site which becomes active only after a failover.

Here are the definitions of some terms used for installing and configuring the MirrorView data source:

EMC SRDF Mirroring data sourceThe EMC Symmetrix Remote Data Facility (SRDF) Mirroring data source controls data access from an SRDF node to EMC Symmetrix device groups. The SRDF data source supports AutoStart application modules (such as the AutoStart modules for Exchange and Oracle). These critical applications depend on databases, resource groups, volumes, and data sources. AutoStart brings these resources online in a controlled sequence when failover occurs. SRDF focuses on disaster avoidance, not disaster recovery.

Term Definition

CLARiiON ID A CLARiiON is identified by an ID, which is called CLARiiON ID.

CLARiiON SP A CLARiiON has two Storage Processors (SPs): SP A and SP B, which are identified by IP addresses. Coupled with a valid username and password, the IP addresses are used for administrating the CLARiiON.

Consistency Group Multiple mirrors are grouped together to operate as a unit for MV/S configurations.

LUN A LUN is a logical unit number, which is used by a host to identify a SCSI device (in CLARiiON). LUNs can also be referred to as an image.

Mirror A mirror contains a primary image and a secondary image. At any time, only the primary image is accessible to the local host. A mirror is identified by user identification (UID).

Navisphere® Manager

A web, browser-based interface for managing MV/S functionality.


Data Sources

Note: In order to use this predefined data source, you must have a valid license for the SRDF data source. For information about getting and entering a license key for this data source, see the “Preface” at the start of this guide.

The EMC SRDF Mirroring data source provides application availability to critical data, even in the event of node or data source failure. The SRDF data source monitors and manages data resources by using EMC replication technology to create a local or remote restrictedly image. If any resource included in the resource group becomes unresponsive or fails during operation, AutoStart automatically retries locally. (Only when absolutely necessary does it retry remotely).

When you have a critical application such as Oracle or MS Exchange that you need to monitor and protect from downtime or server failure, you can configure AutoStart to:

◆ Restart the application on the same server

◆ Relocate the application to a local server

◆ Relocate the application to a remote site where you are using SRDF to replicate your data.

You can also relocate resource groups manually, using options in the AutoStart console. Note, however, that nodeless resource groups are managed by a rule and are typically relocated by an on-demand trigger. For more information, see “Nodeless resource groups” on page 95.

In the AutoStart console when you are configuring a resource group that contains an SRDF data source, make sure the startup and shutdown sequences attach and detach the SRDF data source in the correct sequence.

SRDF Mirroring Data Source tabs in the AutoStart console

An SRDF Mirroring data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the SRDF Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

SRDF/S versus SRDF/A

SRDF/Synchronous (SRDF/S) and SRDF/Asynchronous (SRDF/A) work in many scenarios to protect critical applications from site failure and prevent data loss. The mode of SRDF replication (SRDF/S or SRDF/A) you use depends mainly on three factors:

◆ Recovery time objective (RTO) — The amount of time that you are willing to take to restart critical applications.

◆ Recovery point objective (RPO) — Your tolerance for data loss.

EMC SRDF Mirroring data source 77

Data Sources

◆ Maximum distance — Geographic distance between servers in the network.

You can reduce the RTO by automating the disaster restart procedures and processes, with or without operator intervention.

A typical SRDF configuration

A typical SRDF configuration is an AutoStart domain of at least two nodes connected to a pair of EMC Symmetrix data arrays, as shown in Figure 3.

Figure 3 Typical SRDF configuration

SRDF manages devices as a group, as either:

◆ A device group

◆ A composite group

Mode Recovery time objective (RTO)Recovery time objective (RPO)

Maximum Distance Bandwidth

SRDF/S A few minutes to bring applications online

No data loss Approximately 200 km

High

SRDF/A A few minutes to bring applications online

Seconds to minutes of data loss

Unlimiteda Medium

a. Contact your EMC Customer Support Representative for specific configuration issues.


Data Sources

Example

The EMC SRDF Mirroring data source provides application availability to critical data, even in the event of node or data source failure. A typical configuration is an AutoStart domain of at least two nodes connected to a pair of EMC Symmetrix data arrays, as shown in Figure 4.

Figure 4 SRDF example

The EMC SRDF software controls access within the Symmetrix data array, providing access to one or more devices managed as a device group. The device groups are configured into two sides—the R1 (primary) side and the R2 (secondary) side. When a machine is connected to a pair of Symmetrix data arrays, the side that is its primary connection is always referred to as the R1. Normally, data is mirrored dynamically from R1 to R2. In the event that R1 becomes inaccessible, a failover takes place and the data is then accessible from R2, the secondary connection. When a host is connected to R2, no mirroring is taking place, although the host(s) have read and write access on R2.

The EMC SRDF Mirroring data source controls data access from the node to the Symmetrix. This data source can be included in a resource group to provide high availability in the event that the data source or other resources fail.

The resource group controls node access to the Symmetrix. The data source uses a predefined device or consistency group name to determine which Symmetrix and corresponding devices it should use for connection.

SRDF/Synchronous

SRDF/Synchronous (SRDF/S) is a business continuance solution that maintains a real-time (synchronous) copy of data at the logical volume level in Symmetrix 3xxx, 5xxx, 8xxx, or EMC Symmetrix DMX™ systems in the same or separate locations. A typical SRDF/S configuration looks like the example in Figure 5 on page 80.

Symmetrix SRDF/S offers the following major features and benefits:

◆ High data availability


Data Sources

◆ High performance

◆ Flexible configurations

◆ Host and applications software transparency

◆ Automatic recovery from a component or link failure

◆ Significantly reduced recovery time after a disaster

◆ Increased integrity of recovery procedures

◆ Reduced backup and recovery costs

◆ Reduced disaster recovery complexity, planning, testing, and so forth

Figure 5 SRDF/Synchronous

The SRDF/S operation is transparent to the host operating system and host applications. It does not require additional host software for duplicating data on the participating Symmetrix units.

SRDF/S offers greater flexibility through additional modes of operation, specifically:

◆ Semi-synchronous mode

◆ Adaptive copy-write pending mode

◆ Adaptive copy-disk mode

For more information on these SRDF/S modes of operation, see the discussions of primary and secondary modes of operation in the EMC Symmetrix Remote Data Facility (SRDF) Product Guide.


Data Sources

Example of using AutoStart with SRDF/S solutionConsider a multiple-node (three or more) AutoStart cluster configuration where Host A is running a critical application such as Exchange, the RPO is zero and the RTO is 5 minutes or less (the applications must be available to clients within a few minutes). In this scenario, if SRDF/S is implemented and Host A fails, AutoStart automatically retries starting the application on Host B, also on the primary (R1) site. If Host B fails, AutoStart can use the SRDF data source to failover remotely to the secondary (R2) device. AutoStart then restarts the application automatically on Host C. The remote failover can be done with zero data loss up to 200 km. AutoStart can also automate the failback operation.

If a personality swap is enabled when relocating resources to Host C, synchronization occurs between R1 and R2, and as a result, the identities of the original R1 and R2 are swapped. For more information on personality swap refer to, “Personality swap illustrated” on page 83

SRDF/Asynchronous

Beginning with EMC Enginuity™ Version 5670, the Symmetrix DMX system supports the asynchronous replication product named SRDF/Asynchronous (SRDF/A). SRDF/A provides a point-in-time image on the secondary (R2) device that is only slightly behind the primary (R1) device. SRDF/A session data transfers to the secondary Symmetrix system in predefined timed cycles or delta sets, eliminating the redundancy of multiple same track transfer over the link, potentially reducing the required bandwidth.

SRDF/A provides a long-distance replication solution with minimal impact on performance. This level of protection is intended for customers who require minimal host application impact while maintaining a consistent, restrictedly image of their data on the R2 side. In the event of a disaster at the R1 site or if SRDF links are lost during data transfer, a partial delta set of data can be discarded, preserving consistency on the R2 with a data loss of no more than two SRDF/A cycles.

For more detailed information on SRDF/A, refer to the discussion of SRDF/Asynchronous operations in the EMC Symmetrix Remote Data Facility (SRDF) Product Guide.

The illustration in Figure 6 shows a typical SRDF/A configuration.

Figure 6 SRDF/Asynchronous


Data Sources

The illustration shows that the primary site, the active cycle collects all new writes and tags them as belonging to cycle N. There is also an inactive cycle (N-1) that does not receive new data, but pushes data that was collected when it was the active cycle to the remote side.

At the remote site exists an inactive cycle (N-1), which is receiving data from its partner inactive cycle in the primary site and placing it in global memory. The active cycle (N-2) at the remote site drains the global memory from a previous cycle, received in its entirety, representing a consistent image of the data.

SRDF/A benefits SRDF/A provides the following features and benefits:

◆ Supports extended data replication with database consistency.

◆ Promotes efficient link utilization resulting in lower link bandwidth requirements.

◆ Maintains a consistent point-in-time image on the secondary (R2) devices at all times.

◆ Supports all current SRDF topologies (ESCON, EMC FarPoint™ , point-to-point and switched fabric Fibre Channel, and GigE).

◆ Requires no additional hardware, such as switches or routers.

◆ Supports most open systems hosts and data emulation types supported by the Symmetrix system (FBA and CKD).

For more information on supported hosts, refer to the EMC Support Matrix at http://EMC.com or https://support.EMC.com, or contact your EMC Sales Representative.

◆ Imposes minimal impact on the back-end disk directors

◆ Provides a performance response time equivalent to writing to non-SRDF devices

◆ Allows failover and failback capability between the R1 and the R2 sites

For more information on SRDF/A, refer to the EMC Symmetrix Remote Data Facility (SRDF) Product Guide.

Example of using AutoStart with SRDF/A solutionConsider a multiple-node (three or more) AutoStart cluster configuration where Host A is running a critical application such as Oracle, a tolerance for minimal data loss (30 seconds old, by default), an RTO of 10 minutes, and a very large LAN, for example a regional area like New England. In this scenario if SRDF/A is implemented and Host A fails, AutoStart automatically retries starting the application on Host B, also on the primary (R1) site with zero data loss. If Host B fails, AutoStart can use the SRDF data source to failover remotely (with minimal data loss) to the secondary (R2) site. AutoStart can be configured to restart the application on Host C, either automatically or with operator intervention. The remote failover can be done with minimal data loss over very large (effectively unlimited) distances. AutoStart can also be used to automate the failback operation.


https://Support.emc.com

http://EMC.com

https://Support.emc.com

Data Sources

Personality swap illustrated

Personality swap is a feature provided by Dynamic SRDF which gives the capability of swapping the R1/R2 personality of an SRDF device pair. After a personality swap, the primary R1 devices become secondary R2 devices and the secondary R2 devices become primary R1 devices. The personality swap allows the R2 side to take over operations while retaining a remote mirror on the R1 side. This feature is especially useful after failing over an application from the R1 to the R2 side.

In Figure 7 on page 83 you can see a personality swap that occurs for a Symmetrix Remote Data Facility (SRDF) configuration. In a personality swap, source R1 devices become target R2 devices and target R2 devices become source R1 devices.

Note: Personality swap is available for SRDF/S and SRDF/A for EMC Enginuity version 5x71 and later. For earlier versions of Enginuity, SRDF/S is available.

With the personality swap feature, you can configure your disaster restart process so that after AutoStart performs the SRDF failover operation, the R2 devices are configured as R1 and the original R1 devices become the R2s. You may subsequently test your failover/failback operations with remote protection. This process lets you test your disaster restart processes with minimal downtime and with remote replication.

In the EMC AutoStart dialog box, when the SRDF data source is enabled with personality swap set to Immediate, a delay of two minutes or longer could result before the data source gets attached.

If the source becomes unavailable due to a disaster, dynamic swap is not available.

Figure 7 SRDF personality swap


Data Sources

Device groups

The EMC SRDF software controls access within the Symmetrix data array, providing access to one or more devices managed as a device group. The device groups are configured into two sides:

◆ The R1 (primary) side, which is Site A in Figure 8

◆ The R2 (secondary) side, which is Site B in Figure 8

Figure 8 SRDF device groups

When a host is connected to the primary site Symmetrix, it writes to R1 devices. The R1 has a remote mirror, R2, which is located at the secondary site. In the event that the primary Symmetrix becomes inaccessible, AutoStart initiates a failover and restarts the application off of R2, the secondary connection. The side that is its primary connection is always referred to as the R1. Normally, data is mirrored dynamically from R1 to R2. In the event that R1 becomes inaccessible, a failover takes place and the data is then accessible from R2, the secondary connection. When a host is connected to R2, no mirroring is taking place, although the host(s) have read and write access on R2.

The illustration shows a typical AutoStart three-node cluster that spans two Symmetrix arrays using SRDF/S for remote replication. It shows how AutoStart can be configured to first failover an application or service running on the primary node on Site A to a local node connected to the same Symmetrix. Then, if necessary (for example, the site fails), AutoStart relocates resources remotely to Site B.

When a failover takes place, data can be re-synchronized by an R1 Update and/or personality swap operation. The R1 Update can be initiated manually through the graphic user interface (GUI) or command line interface (CLI), whereas the personality swap is initiated by an attach operation or by the R1 Update. In a personality swap there is an exchange of the RDF personality designation of a specified device group. See “Personality swap illustrated” on page 83.

Note: EMC recommends that you use the personality swap feature rather than the R1 Update SRDF module menu option.


Data Sources

The EMC SRDF data source controls data access from the node to the Symmetrix array. This data source can be included in a resource group to provide high availability in the event that the data source or other resources fail. The resource group controls node access to the Symmetrix. The data source uses a predefined device group name to determine which Symmetrix array and corresponding devices it should use for connection.

Mount points data source for Windows

Note: This data source is available only on Windows.

Windows Mount Points is a new data source supported in AutoStart 5.3 onwards, where support for Mount Points is limited to the shared disk data source:

◆ The Windows mount points data source must be created on a root drive that is accessible from all the nodes.

◆ AutoStart support for mount points is currently limited to the shared disk data source.

◆ Mount points must be created before creating the shared disk, and deleted after the shared disk has been deleted.

◆ In a resource group, mount points must go after the shared disk in the startup sequence, and go before the shared disk in the shutdown sequence.

Note: The root disk must be accessible on all the nodes during creation and deletion of the mount point data source in AutoStart. The root disk must be accessible only from one node in the cluster while attaching and detaching the mount point data source in AutoStart.

Note: Mount points within mount points is not supported.

Before you begin

◆ All disk connections must be set up correctly.

◆ All connected nodes must be online and recognize the disk.

◆ The disk must be identified to the shared disk driver as a shared disk that needs to be managed.

◆ If you are using the data source in a resource group created using AutoStart's Windows Print Services module, refer to “Windows Print Services module” on page 116.

◆ The mount points must be created on the shared disk (using Windows management console) before configuring mount points in AutoStart.

Tabs in the AutoStart console

A Mount Point for Windows data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

Mount points data source for Windows 85

Data Sources

Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

Shared Disk Device for Windows data sourcesWhen you are using shared storage devices, such as shared SCSI or storage area network (SAN) on Windows nodes, or when two or more nodes in the AutoStart domain are physically connected to a single shared disk drive using connections, use the Shared Disk Device for Windows data source in AutoStart to manage it. When you create and use the Shared Disk Device data source for a shared disk device, AutoStart determines the node that has access to the disk to perform read and write operations at any given point in time. It does this using a filter driver that is installed by AutoStart. The filter driver ensures that only one node can read and write to the data source at a time.

A shared disk device for Windows is a storage area network SAN-attached drive or drive array. It can be mounted from different nodes. In creating the data source, you make sure all nodes access the shared disk using one uniform drive letter.

For requirements and prerequisite conditions for EMC shared disk data source for Windows refer EMC AutoStart Administrator’s Guide.

If you are using the data source in a resource group created using AutoStart’s Windows Print Services module, refer to additional information in “Windows Print Services module” on page 116.

A share is captured when the data source attaches for the first time. Once the data source is attached, the AutoStart agent monitors for new and deleted shares; but even then, if you add a new node to the shared disk you must configure it in the data source. How you do this is described in AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source. You can also set up the following:

◆ You can set up AutoStart to warn you when a disk is getting full.

◆ When a disk is no longer going to be shared between nodes, it must be removed from all data sources and then removed as a shared disk. This is either done for you by AutoStart if that's how the data source is set up, or you must do it yourself.

Note: You can include a Shared Disk Device for Windows data source in a Composite data source. For more information, see “Composite data sources” on page 69.

EMC recommends that you configure multiple AutoStart domain lines when using shared disks. If you do not, and if network partition occurs between the machines accessing the disk, communication is lost and there are no checks to ensure that only one machine accesses the disk at a time. Disk corruption may occur due to concurrent writes. For more information, see “AutoStart domain networks” on page 23.

EMC recommends that Windows shared disk configurations use like controllers. Using different controllers may cause unexpected behavior.


Data Sources

Shared Disk Device for Windows data source tabs in the AutoStart console

A Shared Disk Device for Windows data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, the Status tab, and the Disk Management tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table.

VERITAS Volume Manager for Windows data sourcesVERITAS Volume Manager (VxVM) is essentially a software array. If a resource group you are managing needs access to a disk group that is managed by Windows VERITAS Volume Manager (VxVM), you must create a VERITAS Volume Manager for Windows data source for the disk group. A disk group can be made up of multiple volumes, each of which is made up of one or more disks. The operating system sees each volume as a single disk, while VERITAS Volume Manager provides the underlying management of the various disks in each volume of the disk group.

You must configure the disk group using VERITAS’s procedures before you create a data source for it in AutoStart and manage it using AutoStart.

AutoStart's VERITAS Volume Manager for Windows data source manages the VxVM disk group and attaches and detaches the disk group to and from the appropriate node. When AutoStart attaches the disk group to a node, all volumes and their corresponding drive assignments are attached to that node.

Note: You can include a VERITAS Volume Manager for Windows data source in a Composite data source. For more information, see “Composite data sources” on page 69.

VERITAS Volume Manager for Windows data source tabs in the AutoStart console

A VERITAS Volume Manager for Windows data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

VERITAS Volume Manager for Solaris data sourcesVERITAS Volume Manager for Solaris (Sun VxVM) is essentially a software array. The VERITAS Volume Manager for Solaris (Sun VxVM) data source is a pre-configured VERITAS Volume Manager disk group. The VERITAS Volume Manager disk group must be configured using the procedures supplied by VERITAS before the disk group can be managed by AutoStart.

A disk group can be made up of multiple volumes, which in turn are made up of one or more disks. The operating system sees each volume as a single disk, while VERITAS Volume Manager provides the underlying management of the various disks in each volume of the disk group. VERITAS Volume Manager objects cannot span disk groups; that is, all the operations on a disk group are confined to that particular group.

VERITAS Volume Manager for Windows data sources 87

Data Sources

A disk group enables high availability because it can be shared by two or more hosts but can be accessed by only one host at a time. For example, if you have two hosts and shared storage, one host can take over the ownership of the disk groups and drives if the other host fails.

The data source manages the disk group and attaches and detaches it to and from the appropriate node. When the disk group is attached, all volumes are attached.

Using in a resource group

When you include a VERITAS Volume Manager for Solaris data source in a resource group, you must also make sure that the various file systems that are mounted on the volumes of the disk group are managed by AutoStart using the Solaris File System data source (which is described in “UNIX file system type data sources” on page 92). In the resource group's startup and shutdown sequences, the VERITAS Volume Manager for Solaris data source must attach and detach in the following ways:

◆ Attach sequence: The VERITAS Volume Manager for Solaris data source must attach before the file system data source residing on the volumes attaches. This means that in the resource group's startup sequence, VERITAS Volume Manager for Solaris data source must be listed before the Solaris File System data source.

◆ Detach sequence: The VERITAS Volume Manager for Solaris data source must detach after the file system data source residing on the volumes attaches. This means that in the resource group's shutdown sequence, VERITAS Volume Manager for Solaris data source must be listed after the Solaris File System data source.

VERITAS Volume Manager for Solaris data source tabs in the AutoStart console

A VERITAS Volume Manager for Solaris data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

Windows Network Share data sourcesAutoStart provides two data source types to connect to Network Attached Storage (NAS) units or other nodes that share their file systems over the network. One type—the Windows Network Share data source—is for nodes running Windows.

Note: The second type—the Network File System data source—is used on nodes running a supported version of UNIX. See “NFS data sources” on page 91 for more information.

A Windows network share is an advertised share from a Windows storage server or other NAS device. A Windows network share gives users access to the content of a remote disk over a network. The network share's content may be the entire remote disk, or a portion of the disk. You can make a Windows network share highly available by creating a data source for it.


Data Sources

With AutoStart’s Windows Network Share data source, you can manage access to network shares defined in a Windows domain or workgroup from nodes in the AutoStart domain. The data source lets you access a network share on a machine that is inside or outside the AutoStart domain, and attaches the share to a node in the AutoStart domain. When AutoStart attaches a Windows Network Share data source to a node, it maps a network drive on the node it attaches the data source to. The network drive is identified by the drive letter you designate for it when you configure the data source.

Note: AutoStart manages network attached data sources using standard OS operations.

While the data source is attached to a node, users can access the share and use it as a normal network drive. Make sure the Windows network share is accessible from each node where the data source will attach.

Note: About detaching: Because AutoStart uses the System user to assign the drive for a Windows network share, rather than the logged in user, only AutoStart can detach a data source for a Windows network share that it has attached. You cannot detach the data source from your desktop or disconnect the drive. Standard users, and even administrators, do not have privileges to disconnect the drive. AutoStart removes the drive from the desktop when the data source detaches. Any connected data source detaches if the node reboots.

Windows Network Share data source tabs in the AutoStart console

A Windows Network Share data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

AIX Volume Group data sourcesThe AIX Volume Group data source manages an AIX Logical Volume Manager (LVM) volume group. Before you can create a data source for an AIX Volume Group and manage it using AutoStart, you must configure the AIX Volume Group by using standard procedures.

An AIX volume group is made up of multiple volumes; in turn, each volume is made up of one or more disks. The operating system sees each volume as a single disk, while the LVM manages of the various disks in each logical volume of the volume group.

The AIX Volume Group data source manages the volume group and attaches and detaches it to and from the appropriate node. When AutoStart attaches the volume group, all volumes in the group are attached.

Requirements for an AIX Volume Group data source

If you are going to run an AIX Volume Group data source in a resource group:

◆ You must make sure that the file systems that are mounted on the logical volumes of the volume group are also managed by AutoStart, using AutoStart's AIX File System data source (see “UNIX file system type data sources” on page 92).

AIX Volume Group data sources 89

Data Sources

◆ The AIX Volume Group data source must attach before any AIX file system data source that resides on the volumes attaches. For that reason, in the resource group configuration, make sure the AIX Volume Group data source is listed in the resource group's startup sequence before the AIX File System data source. For more information, see “Startup and shutdown sequences” on page 100.

◆ Make sure the volume groups are configured so that they do not activate automatically at system startup.

◆ When a physical volume in an AIX volume group changes, you must reflect the change in AutoStart.

AIX Volume Group data source tabs in the AutoStart console

An AIX Volume Group data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

HP-UX LVM data sourcesThe HP-UX LVM data source is a data source that manages HP Logical Volume Manager (LVM) volume groups.

You must configure the volume group using standard procedures before the volume group can be managed by AutoStart.

A volume group can be made up of multiple volumes, which in turn are made up of one or more disks. The operating system sees each volume as a single disk, while the Logical Volume Manager provides the underlying management of the various disks in each volume of the volume group.

The HP-UX LVM (Logical Volume Manager) data source manages the volume group and attaches and detaches it to and from the appropriate node. When AutoStart attaches the volume group, all volumes are attached.

If you are using the HP-UX LVM data source in a resource group, the various file systems that are mounted on the volumes of the volume group must also be managed by AutoStart using the HP-UX File System data source. The HP-UX LVM data source must be attached before the HP-UX File System data sources that reside on the volumes is attached. In the resource group configuration the HP-UX LVM data source must be listed before the HP-UX File System data source.

By default, HP-UX LVM data source brings up all volume groups automatically at machine startup. For that reason, if a volume group is being managed by AutoStart, you must configure the system so that it doesn’t automatically start up the volume groups managed by AutoStart.


Data Sources

HP-UX LVM data source tabs in the AutoStart console

An HP-UX LVM data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

NFS data sourcesAutoStart provides two data source types to connect to Network Attached Storage (NAS) units or other nodes that share their file systems over the network. One type—the Network File System (NFS) data source—is used on nodes running a supported version of UNIX.

Note: The second type—the Windows Network Share data source—is for nodes running Windows. For more information, see “Windows Network Share data sources” on page 88.

NFS is a UNIX or Linux file system. The NFS data source manages NFS volume groups. NFS provides transparent access to files on remote hosts as if the files were local. The client side uses a remote file system to access an NFS server that provides file data. You must configure the network file system using standard procedures before it can be managed by AutoStart.

Note: AutoStart manages network attached data sources using standard OS operations.

NFS data source tabs in the AutoStart console

An NFS data source’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

Sun Solstice DiskSuite data sourceThe Sun Solstice DiskSuite (SDS) data source is a Solaris-only data source that uses a pre-configured Solstice DiskSuite disk set. The disk set must be configured using SDS procedures before the disk set can be managed by AutoStart.

Each server must have an Sun Solstice DiskSuite default replica database residing on its local disk. Once the default replica database is created, a disk set can be defined.

A disk set contains a set of logically grouped UNIX devices called metadevices, which can be shared between two servers. The operating system sees each metadevice as a single disk, which can then be mounted as a file system. Sun Solstice DiskSuite provides the underlying management of the disk set, managing each of the metadevices.

The data source manages the disk set and oversees attaches and detaches to and from the appropriate node, ensuring that only one node is attached to the disk set at any one time. When the disk set is attached, all metadevices in the disk set definition are attached.

NFS data sources 91

Data Sources

When you are using the Sun Solstice DiskSuite data source in a resource group, you must also make sure that the various file systems that are mounted on the volumes of the disk set are managed by AutoStart using the Solaris File System data source. The Sun Solstice DiskSuite data source must be attached before the file system data source residing on the volumes are attached. In the resource group configuration the Sun Solstice DiskSuite data source must be listed before the Solaris File System data source.

Sun Solstice DiskSuite data source tabs in the AutoStart console

A Sun Solstice DiskSuite data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data Source table. Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for information about using this data source.

UNIX file system type data sourcesUNIX file system data sources are available on:

◆ AIX for a JFS file system

◆ HP-UX for an HFS or VxFS file system

◆ Linux for an ext2, ext3, hpfs, or reiser file system

◆ Solaris for a UFS or VxFS file system

Refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for steps for configuring them.

IMPORTANT

When creating a UFS data source you are asking AutoStart to manage it. Therefore you must configure these file systems to not be automatically mounted at system boot time. For example, on Linux you must set the noauto option in Linux’s /etc/fstab.

Tabs for UNIX file system type data sources in the AutoStart console

A UNIX file system data source's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Advanced tab, and the Status tab. You can see a list of all data sources in the AutoStart domain by viewing the Data source table.


CHAPTER 5Resource Groups

This chapter describes AutoStart resource groups that monitor your applications:

◆ Resource groups ..................................................................................................... 94◆ The preferred nodes list .......................................................................................... 95◆ Resources in a resource group................................................................................. 99

Resource Groups 93

Resource Groups

Resource groupsA resource group is basically a collection of everything you need to restart an application on another node. A resource group is made up of all the resources that must be restarted as a group in order to relocate an application. With this collection of resources you provide the sequences in which the resources must be shut down when bringing the resource group offline and started when bringing the resource group online. Together, the resources and the startup and shutdown sequences are a logical group that you can monitor and manage as a single entity or application.

The application components that you must add to a resource group are managed resources, and they include:

◆ Processes

◆ Services

◆ Data sources

There are additional management components you’ll need to define for the resource group. You'll need to define:

◆ A virtual IP address, also called a managed IP address, which you’ll add to the resource group

◆ A node alias if needed for the resource group

Once you define the resource group, AutoStart treats it and all of the resources defined in it as a package. Resources in the resource group always run on the same node. If one of the resources in the resource group cannot run properly on a node, the whole resource group relocates to the next node in the preferred node list. You can also add customization using:

◆ Scripts

◆ Timing delays

◆ Event messages sent to the error log

◆ Utility processes

◆ Failback options

To configure a resource group you must create the resource group, select its preferred nodes, select its resources and specify the resource group’s startup and shutdown sequences, and configure its options on the Options and Advanced tabs.

Resource groups for Microsoft IIS sites

If you want to create a resource group that monitors your Microsoft IIS sites, AutoStart provides a wizard that you use for creating such resource groups. Once the wizard has created the resource group, you can manage the IIS-site resource group as you would any other resource group. For more information, see “Microsoft IIS module” on page 115.


Resource Groups

Modules

AutoStart has modules that you can use for quickly creating resource groups for some of the more commonly-used applications. A module consists mainly of a wizard that creates a module instance, which behaves like a resource group that you can manage using AutoStart. The modules are essentially plug-ins that you can enable using license keys, which you must obtain from EMC Product Licensing. For more information, see “AutoStart modules” on page 104.

Resource group tabs in the AutoStart console

A resource group's AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Options tab, the Advanced tab, the Availability Tracking tab, and the Status tab. You can see a list of all of an AutoStart domain's resource groups by viewing the resource group table.

The preferred nodes listA resource group’s preferred nodes list (also called the valid nodes list) identifies the nodes within the AutoStart domain where the resource group’s managed resources can run. The preferred nodes list can include all of the nodes in the AutoStart domain, or it can be a subset, but it must include at least two nodes. Each of the nodes in the preferred nodes list must be configured with the services, processes, and directories required to run the resource group.

Note: Some options mentioned in this topic are not available in AutoStart module wizards or in module instance tabs in the AutoStart console. To set such options for a module instance, you can use the AutoStart console's resource group tabs, but only if you are in advanced user mode.

Nodeless resource groups

You are not required to set up a preferred nodes list for all of your resource groups. A resource group configured without a preferred node list is called a nodeless resource group. AutoStart manages a nodeless resource group in the same way it manages a resource group that has a preferred nodes lists; the big exception is that a nodeless resource group does not have a specified relocation destination. So, how does AutoStart know where to relocate a nodeless resource group to? The answer is that you must create a rule that will control the relocation of the resource group when a failure occurs. A nodeless resource group can be started only by the rule that you've created for it. The nodes that such a resource group can relocate to are also determined by the rule. For more information, see “Rules” on page 134.

You cannot start a nodeless resource group from the AutoStart console nor from the command line interface (CLI). You can relocate a nodeless resource group manually only by using an on-demand trigger (described in “On-demand triggers” on page 147).

The preferred nodes list 95

Resource Groups

Sequence is important

In the preferred nodes list, you sequence the nodes in the order in which you want AutoStart to attempt to bring the resource group online if failover occurs, starting with the top-most node in the list, working toward the node at the bottom of the list. The sequence of nodes is important only if you want AutoStart to automatically relocate the resource group when the node where it is running becomes unavailable.

The first node, at the top of in the list, is the most-preferred node, meaning that, if at all possible, resources in the resource group should run on this node. If that node is unavailable, AutoStart moves the resource group to the next node in the list, and so on. (AutoStart can fail the application back to the preferred node if you select the Auto Failback to First Node in Preferred Node List checkbox on the resource group's Options tab.

Nodes are available to all resource groups

A node can be included in any number of resource groups. This lets you implement various availability solutions for the nodes and the applications on them. For more information, see “Nodes” on page 36.

Balance the load

When you configure your resource groups' preferred nodes lists, you want to make sure the load is balanced when failover occurs. Because you specify the sequence of nodes on which AutoStart will try to bring each resource group online, you can configure a strategy where multiple resource groups running on the same node do not relocate to the same node in the event of failure. Instead, each resource group can relocate to a different node, preventing the possibility of overloading the destination nodes. To achieve this, specify a unique node sequence for each resource group so that when a node fails or you bring it down for maintenance there’s a reasonable load balance. What you want to avoid is creating preferred nodes lists that cause applications to indiscriminately relocate to a single node, which could cause performance degradation.

Create an alternate list of nodes

You can also create, essentially, alternate preferred nodes list. You do this by adding a node group separator to the preferred nodes list; doing this creates multiple node groups. In AutoStart 5.4.x you can have two or more node groups.

Any nodes that you place below the node group separator are nodes that an operator can fail the application over to, but that AutoStart cannot automatically fail the application over to (for example, nodes on the other side of a WAN link). Use this separator to prevent automatic failover to specific nodes, while retaining the option of allowing an AutoStart user the option to relocate the resource group to the nodes manually. For more information, see “Node groups” on page 98.

If you use the node group separator in the preferred nodes list, you must also consider how you want to configure the Auto Node Group Failover and Notify on Node Group Failover checkboxes on the resource group’s Options tab.


Resource Groups

Note: When creating a resource group for use with a data source for SRDF Mirroring, you must use node groups for a different purpose, which is to map nodes to the data source's R1, R2 and R21 hosts. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: When creating a resource group for use with an EMC MirrorView Mirroring data source, you must use node groups as described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Use a rule to determine the node

Instead of using preferred nodes lists, you can set up resource groups so they are managed by a rule that determines the nodes a resource group relocates to when there is a failure. If this is how you choose to set up a resource group, then you create a nodeless resource group. In this case, you do not configure a node list at all. Instead, you write a rule to determine the most suitable location for the resource group if node failure occurs. For more information, see “Nodeless resource groups” on page 95.

You can retrieve a list of a resource group's preferred nodes list for use in a rule by using the ft::GetGroupNodeList subroutine.

Setting up a preferred nodes list

To set up a list of a resource group's preferred nodes:

1. Go to the Settings tab while creating or editing a resource group.

2. Double-click the nodes in the Available Nodes list to move the nodes to the Selected Nodes list.

3. You can add a node group separator to the Selected Nodes list to create two node groups. For more information, see “Node groups” on page 98.

4. To change the sequence of the selected nodes, move the nodes by selecting the node name or node group separator in the Selected Nodes list and clicking the up and down arrow buttons.

5. Click Apply when you are done.

Limits when using AutoStart modules

There is a limit to the number of nodes a module instance can have in its preferred nodes list; you can use no more than:

◆ 5 nodes for a Microsoft Exchange 2003 module instance

◆ 3 nodes for a Windows SQL Server 2005 module instance

◆ 3 nodes for a Oracle module instance

◆ 2 nodes for a Windows Print Services module instance

The preferred nodes list 97

Resource Groups

Node groups

You can set up as many as two preferred nodes lists for a resource group. You do this by using the node group separator in the preferred nodes list when you configure the resource group using its Settings tab. On the Settings tab is an entry called Node Group Separator, shown in Figure 1.

Figure 1 Node groups displayed in the AutoStart console

All nodes above the separator comprise the first node group AutoStart is to use for relocating the resource group; all nodes below the separator comprise the second node group AutoStart is to use for relocating the resource group.

Assuming you are going to set up a preferred nodes list for your resource groups, you’ll need to consider using node groups. By using the node group separator in the preferred nodes list, you create two sets or collections of nodes, called node groups. When AutoStart tries to relocate a resource group to another node, it first tries the resource group's startup sequence on all nodes in the current node group. Only when all nodes in the current node group are exhausted does it attempt to run the startup sequence on a node in the other node group.

Use node groups to isolate nodes that would be time-consuming or resource-intensive to relocate resources to when, for example, a failure would require relocating resources to a node on the other side of a WAN.

Note: When creating a resource group for use with a data source for SRDF Mirroring, you must use node groups for a different purpose, which is to map nodes to the data source's R1 and R2 hosts. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: When creating a resource group for use with an EMC MirrorView Mirroring data source, you must use node groups as described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Configuring node group failoverBy default, any resource group that you create is set up to require you or another administrator to manually intervene before AutoStart relocates a resource group to the nodes in the second node group. Your manual intervention prevents AutoStart from undertaking a possibly intensive procedure when you know that the entire node group will soon be back online.


Resource Groups

You can, however, override this default. You can configure the resource group so that it fails over to the second node group automatically; you do this by putting a checkmark in the Auto Node Group Failover option on the resource group’s Options tab. Selecting this option will allow AutoStart to use the second node group as the preferred nodes list after all the nodes in the first node group have been exhausted.

You can also use the resource group’s Options tab to specify that you want email notifications to be sent automatically whenever AutoStart fails over the resource group to another node group.

Options to configure for a resource group that has node groupsOn the resource group's Settings tab:

◆ Configure the preferred nodes list to contain two groups of nodes separated by the node group separator. The first node group is the preferred node group, and the second is the alternate. The first node in each node group is that node group's most preferred node, that is, the node where you would prefer the resource group to run.

On the resource group’s Options tab:

◆ Consider the Auto Node Group Failover checkbox.

◆ Consider the Notify on Node Group Failover checkbox.

Resources in a resource groupThe resources that you add to a resource group can consist of any or all of the following:

◆ Services

◆ Node aliases Processes

◆ Managed IP addresses

◆ Data sources; and in the startup and shutdown sequences, data source checkpoints, which wait for all proceeding data sources to transition to the Attached state, and are relevant only when using parallel attached data sources. For more information and examples, see “Parallel attach of data sources” on page 64.

◆ Utility processes

◆ Small Perl scripts

◆ Additional informational event messages that are written to the event log

◆ Delays help you ensure that a resource has adequate time to start up or shut down before the next resource starts up or shuts down

You must configure each resource before you add it to a resource group. Remember to give your resources names that make them easy to identify.

Resources in a resource group 99

Resource Groups

No manually controlled resources allowed

You decide which resources make up a resource group. They should be resources that you want AutoStart to manage for you. Do not add any resources that you want to control manually. If you do, when the resource group is brought online, resources that have been brought online manually may cause the resource group to behave unpredictably. For example, the resource group may initiate a failover to another node.

Resources in multiple resource groups

Do not use a resource in two resource groups that run at the same time. If AutoStart attempts to assign the same resource to more than one node, it takes down the resource on the first node and assigns it on the second. When the first node notices the failure, it repeats the procedure. The two resource groups continue to compete for the common resource without resolution.

From version 5.5.0, the same resource cannot be added to multiple resource groups. If the same resource is already present in multiple RGs, which may happen in case the RG was upgraded from previous Autostart versions, the RG cannot be modified unless domain level attribute FT_RG_AMR is set to 1. Once RG is modified, FT_RG_AMR can be reset or set to 0 to prevent same resource from being added to multiple RGs. Optionally, the attribute can be permanently set to 1. The RG modification is allowed only through GUI console and not from ftcli.

If an application must run on multiple nodes at the same time, define a unique process service definition for the application to AutoStart and define multiple resource groups. Then, add to each of the two resource groups one of the application’s process instances and any other resources needed by the processes. If multiple instances of an application can potentially run on the same node, ensure that multiple instances of the application can run on the same node at the same time. Otherwise, each resource group must have a unique node list to prevent them from executing concurrently.

Startup and shutdown sequences

When a resource group is brought online, the startup sequence executes. Each resource is brought online in the order listed in the startup sequence. If one of these steps fails, the startup sequence stops and the shutdown sequence is executed to bring the resource group back to a known offline state.

When you define a resource group, you add the resources that are key to the resource group's availability to run. There is no limit to the number of resources that can be in the startup sequence. However, more time will be needed for the resource group to come online if its startup sequence is long. (Note that you can abort the execution of a resource group’s startup sequence by using the AutoStart console.)

Note: Using modules? If you have a module instance that was created using a module wizard, you can see it as a resource group only if you use the AutoStart console as an advanced user. If you do not have this access, you cannot review the startup and shutdown sequences that the wizard created. For more information, see AutoStart online help or the EMC AutoStart Administrator’s Guide.


Resource Groups

When you bring a resource group online, the startup sequence executes. Each resource is brought online in the order listed in the startup sequence. If one of these steps fails, the startup sequence stops and the shutdown sequence is executed to bring the resource group back to a known offline state.

When you take a resource group offline, the shutdown sequence is executed. Each resource comes offline in the order listed in the shutdown sequence. During relocation, the shutdown sequence runs on the source node and then the startup sequence executes on the destination node.

Guidelines and strategies for configuring startup and shutdownOnce you’ve defined the resource group’s resources, then specify the sequence in which those resources are to be started up. Similarly, you also specify the sequence in which the resource group's resources are taken offline. This lets you configure the resource group to operate in the way that best suits your environment.

The sequence in which resources start is basically determined by the dependencies of the resources. For example, you want to attach data sources before starting the services that access the data in those data sources. Ultimately though, it is thorough testing of your startup and shutdown sequences that determines the correct order of resources in a resource group.

Checklist for configuring startup and shutdownYou can also add hooks to the startup and shutdown sequences that:

◆ Delay an action for a specific amount of time.

◆ Launch an auxiliary program to perform an operation such as testing readiness of a component.

◆ Include scripts for performing application-specific activities.

◆ Send messages to the event log.

◆ Provide parallel attach of data sources. For more information and examples, see “Parallel attach of data sources” on page 64.

Pay close attention to instructions for configuring the resources you are working with. There are some sequences that you are required to configure, and the instructions for those resources will tell you what they are. For example, when you are configuring a resource group that contains an SRDF data source, make sure the startup and shutdown sequences attach and detach the SRDF data source in the correct sequence:

◆ In the startup sequence, the SRDF data source must attach before data sources for volume groups, file systems, and shared disks attach.

◆ In the shutdown sequence, the SRDF data source must detach after data sources for volume groups, file systems, and shared disks attach.

Using delays, checkpoints, and parallel attaches in startup and shutdown sequencesThe sequence of resources in a startup sequence determines the order in which AutoStart brings the resource group’s objects online. Likewise, the sequence of resources in a shutdown sequence determines the order in which AutoStart takes the resource group's

Resources in a resource group 101

Resource Groups

objects offline. The sequence is, in essence, a to do list for AutoStart; it is the definitive list of instructions that AutoStart follows when it brings an application online or takes it offline.

Note: This topic describes delays in the context of a startup sequence, but its principles apply to shutdown sequences, too, and are no less important for you to consider when you configure a resource group's shutdown sequence.

Once you know the data sources, IP addresses, processes and services, you need to consider the timeframes in which they are brought online. Some of the resources in a startup sequence may need to be completely online before AutoStart brings the next resource in the sequence online, but other resources may need to go online simultaneously. In essence, you have to add "traffic cops" to the startup sequence to control timing. Your goal in doing this is to make sure that no resource is brought online until all of its dependencies are online.

To regulate the timing of resources, you can:

◆ Delay the startup or shutdown of a process or service.

◆ Wait for a data source to be brought completely online before the next resource in the sequence is brought online, by using a data source checkpoint.

◆ Attach two or more data sources simultaneously, or in parallel.


CHAPTER 6Modules

This chapter describes the applications for which AutoStart provides modules. A module is a wizard that creates a resource group that monitors an application. If a module is not available for an application, you create the resource group yourself, or you can hire EMC professional services to help you:

◆ AutoStart modules ................................................................................................ 104◆ SQL Server 2005 module ...................................................................................... 105◆ Microsoft Exchange 2003 module ......................................................................... 109◆ Microsoft IIS module ............................................................................................. 115◆ Windows Print Services module ............................................................................ 116◆ Oracle module for Windows .................................................................................. 118◆ Oracle module for UNIX ......................................................................................... 122

Modules 103

Modules

AutoStart modules

Note: In addition to the modules described in this chapter, AutoStart contains a component that enables integration with EMC DiskXtender®. Instructions for using this component are provided in the EMC DiskXtender 6.2 Installation Guide. To get a copy of this guide, see “Related documentation” on page 3.

Typically, an AutoStart module is a wizard that can quickly and easily help you to set up a module instance that will operate with a software product that is available in your shop. AutoStart modules are available with an EMC product license key. The modules that are available are:

◆ Microsoft SQL Server 2005

◆ Microsoft Exchange 2003

◆ Microsoft Windows Print Services (No license key is required)

◆ Oracle for UNIX

◆ Oracle for Windows

◆ Microsoft Internet Information Server (IIS) (No license key is required)

◆ Microsoft SQL Server 2000 — Available to preexisting installations only; see the documentation for AutoStart version 5.2 for help using this module. Refer to the Preface of this guide for help getting AutoStart 5.2 documentation.

Note: All modules are available on Windows; Oracle is available on UNIX.

When you use an AutoStart module to make an application highly available, you get:

◆ A wizard that helps you to create the application's resource group

◆ Predefined startup and shutdown sequences along with resource group definitions, which are standard for the application and easier and faster to deploy and support

◆ Custom-defined rules that automate responses and resource handling

◆ The ability to monitor the application’s resources, via the AutoStart console, designed specifically for monitoring and managing the application

◆ Online help that tells you how to use it

Granting user access if you use modules

If you use modules, users who have user-level or operator-level user access must be granted write permissions to the AutoStart console directory, which is: InstallDir\consoleReleaseNumber. This is required so the users can see all of the AutoStart modules in the AutoStart console.


Modules

EMC does not support customized module instances

Because EMC does not support customized module instances, if you customize them it is at your own risk. Instead, EMC recommends that you create a resource group that you can customize to your specific needs. If you choose to customize a module instance, in the AutoStart console you will have to work in Advanced mode.

API subroutines are designed specifically for use with resource groups, not module instances. There are no subroutines that work with module instances. This is because, by definition, module instances create an instance that offers no opportunity for customization, which is the sole purpose of an API subroutine. If you need to customize how AutoStart fails over an application, then you must create a resource group, then customize the resource group using the API subroutines available in AutoStart for customizing resource groups. For more about APIs, see Chapter 8, “Rules, Triggers, and Programming Tools.”

SQL Server 2005 module

Note: This feature is available only on Windows.

The AutoStart module for SQL Server 2005 provides start, stop, SQL services monitoring, and availability tracking. AutoStart continually monitors both Microsoft SQL Server and the node it runs on. If SQL Server should fail, AutoStart tries to restart it; if SQL Server cannot be restarted, AutoStart relocates it to another node.

A mirrored or shared disk provides access to the database data from both nodes. You may need more than one mirror or shared disk if your database log or data files are maintained on more than one volume. For examples of two ways you can configure SQL Server 2005, see “Failover configurations for the SQL Server 2005 module” on page 107.

Note: For up-to-date information, always refer to the most recent AutoStart Release Notes and AutoStart Installation Guide that are available for the version of AutoStart you are running. You can download AutoStart documentation from the Internet. For details, see the “Preface” of this guide.

Failure detection

Failure detection is provided for nodes within the AutoStart domain for processes and services. When a node or application failure occurs, the software senses the failure and initiates recovery. Failure detection is configured using a system of heartbeats to determine the state of the node within the domain. Each node sends heartbeats, and primary agents receive these heartbeats for failure detection. When a process or service fails the SQL resource group is relocated (that is, failed over) to the next node in the preferred nodes list.

Failover

Failover occurs when one of the following fails:

◆ A resource in the SQL Server resource group (resources include the managed IP address, node alias, and data sources)

SQL Server 2005 module 105

Modules

◆ The production node that runs the resource group

When such a failure occurs, the SQL Server services and the other resources in the resource group automatically restart on the next host in the preferred nodes list.

Note: In a WAN environment, you can set up your SQL Server instance so that, in the event of a total production site failure, data is relocated from the production site to a remote site and the SQL server resources are restarted on a host at the remote site. You implement this setup by inserting a node group separator between the production nodes and the remote nodes in the instance's preferred nodes list, and then set additional options (which enable automatic failover and email notification) for relocating over a WAN. For a detailed description of these options, see “Node groups” on page 98.

Failback

When the original nodes or production site are running and available again, the SQL server instance's resources can relocate back to the original or production site; this is called failback. Failback can be a manual process or you can set it up to happen automatically. The resource group's automatic failback option causes the SQL Server resource group to always try to go online on the first node in the preferred node list whenever that node is running. The option is called the Automatic Failback to First Node in Preferred Node List checkbox, and it is available on the resource group’s Options tab.

What to do

Follow these steps to configure SQL Server 2005 to be managed by AutoStart:

1. Make sure you meet the SQL Server module requirements.

2. Follow the installation steps for one of the following SQL Server configurations:

• Installing SQL Server 2005 (this is a normal, active/passive configuration)

• Installing an active/active SQL Server 2005 configuration

• Installing native SQL Server 2005 replication

• Upgrading to MS SQL Server 2005 from a SQL Server 2000 module instance

Instructions for each of these configurations is provided in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: You are not required to upgrade a SQL Server 2000 module instance that manages an instance of SQL Server 2000. If you choose to maintain an instance of the SQL Server 2000 module in AutoStart 5.4.1, note that if you need documentation for the SQL Server 2000 module instance, you must refer to the EMC AutoStart Administrator's Guide for version 5.2 of AutoStart. For instructions on how to get a copy, see the Preface to this guide.


Information for a SQL Server resource group or module instance is provided in the following tabs in the AutoStart console:


Modules

◆ Module instances — You can view the following SQL Server module instance tabs: Settings tab, Resources tab, Notification tab, Status tab, and Availability tab. You can see a list of all instances of SQL Server by viewing the SQL Server module's Summary tab. You can see license information and module nodes by viewing the SQL Server module's Configuration tab.

If you upgraded from AutoStart 5.2 when you had a SQL Server 2000 module instance in place, you will see two versions of the SQL Server module displayed in the console's Modules tree. Online help and AutoStart documentation describes the SQL 2005 module only; for help with the SQL 2000 module, refer to the documentation for AutoStart 5.2.

◆ Resource groups — If you are using the AutoStart console as an advanced user, you can view the following SQL Server 2005 resource group tabs: Settings tab, Options tab, Advanced tab, Availability Tracking tab, and Status tab. You can see a list of all resource groups in the domain by viewing the resource group table.

Note: Some options on the resource group tabs are not available on the module instance tabs.

Failover configurations for the SQL Server 2005 module

The AutoStart module for SQL Server is designed to provide high availability for network clients accessing SQL Server databases. Two types of failover can be configured: active/passive and active/active. Each type is described below.

Active/passive configuration with EMC MirroringAn active/passive configuration, like the one shown in Figure 1, allows you to have a single SQL Server instance running on one of the physical servers in the cluster. The other nodes in the cluster are in standby mode until a failure on the active node or a manual failover during maintenance occurs. Only one SQL Server 2005 virtual server is installed on an active/passive SQL Server cluster environment.

Figure 1 Active/passive configuration for the SQL module

SQL Server 2005 module 107

Modules

In an active/passive configuration, one copy of SQL Server database data is installed on a drive shared by two servers. Only one node is running a database server, while the other node acts as a standby. Under normal conditions, the secondary server has no access to the shared drives. Only in a failover situation does the secondary server take over the SQL Server files and data on the shared drive. SQL Server executable files are installed locally.

The network client computers connect to SQL Server through an alias name or virtual IP address. The alias name or virtual IP address is moved to the server that is currently online. The databases can be located on a shared disk or mirror volumes.The mirrored disks must be of similar size, must not have an operating system or pagefile.sys file on them, and should be dedicated solely to database data.

Active/active configurationIn contrast to active/passive configurations, the active/active configuration offers the advantage of using both servers, while still maintaining a failover solution. An example is shown in Figure 2 on page 108.

This configuration uses two or more separate physical drives, each with a copy of SQL Server database data. Under normal conditions, the primary server for each virtual server views and accesses SQL Server files and data, and performs work. The backup server for each virtual server views files and data only on the shared drive it currently owns. Client computers connect to each SQL instance through a unique alias name or virtual IP address.

An active/active configuration provides multiple SQL Server instances running on both nodes of a two-way cluster. If one of the SQL Server instances in the cluster fails, the failed instances of SQL Server automatically relocate to the other server. This means that both instances of SQL Server run on one physical server, instead of two separate servers.

Figure 2 Active/active configuration of the SQL module

Multi-node SQL Server replicationYou can also use the SQL Server module with multi-node SQL Server replication.


Modules

Microsoft Exchange 2003 module


The EMC AutoStart module for Exchange 2003 version 2.0.15 is an optional add-on that you can use in an AutoStart 5.3/5.4 domain. Also called the Exchange 2003 module, it supports multi-node configurations, and it provides wizards you can use to create module instances that:

◆ Provide WAN high-availability, recovery management, and local failover of Microsoft Exchange 2003 servers and services to a standby node.

◆ Protect your email infrastructure from disasters by continually monitoring Microsoft Exchange Server 2003 and the nodes on which MS Exchange runs.

◆ Simplify how you test disaster restart processes for MS Exchange 2003 and maintain MS Exchange 2003 servers.

With the Exchange 2003 module, the following are supported:

◆ Only active/passive configurations are supported by this module. This limits Microsoft Exchange services to running on one node at a time.

◆ You must use Microsoft Exchange Server 2003 in native mode; mixed mode is not supported for the Exchange 2003 module.

◆ Only back-end servers are supported with this module. Clustered front-end servers (including bridgeheads) are not supported by Microsoft, so make sure you do not use them with this module.

◆ With this module, you can access public folders using Microsoft Outlook 2003 and Microsoft Exchange System Manager.

Note: Nodes in an Exchange 2003 module instance must belong to a Windows domain. You cannot use the Exchange 2003 module on nodes in a Windows workgroup.

The resource group

The Exchange 2003 module provides you with wizards and procedures that package MS Exchange’s services, data sources, and managed IP addresses into a logical group, called a resource group, which AutoStart can monitor and you can manage as a single entity. This resource group provides a highly available MS Exchange Server 2003 instance running on a node.

Once you’ve used the wizards to create the Exchange 2003 resource group, AutoStart treats the resource group like any other. In the event of a failure, AutoStart restarts MS Exchange services in the resource group on another managed node; if the services continue to fail, or the entire node fails, AutoStart relocates the MS Exchange services, data source, and network addresses to a healthy node. You can even configure AutoStart to send email notifications when a failure occurs.

Microsoft Exchange 2003 module 109

Modules

Virtual nodes

The Exchange 2003 module gives you a virtual node solution for clustering Microsoft Exchange Server 2003. The MS Exchange mail server that is currently active appears on the network as a virtual host name used in mailbox administration. This allows AutoStart to relocate MS Exchange services without readdressing the mailboxes. This feature creates much faster failover times and eliminates error-prone Active Directory searches and updates.


Information for an Exchange resource group or module instance is provided in the following tabs in the AutoStart console:

◆ Resource groups — If you are using the console as an advanced user, you can view the following resource group tabs: Settings tab, Options tab, Advanced tab, Availability Tracking tab, and Status tab. You can see a list of all resource groups in the domain by viewing the resource group table.

◆ Module instances — Whether or not you are using the console as an advanced user, you can view the following Exchange module instance tabs: Settings tab, Resources tab, Notification tab, Status tab, and Availability tab. You can see a list of all instances by viewing the Exchange Summary tab. You can see license information and module nodes by viewing the Exchange Configuration tab.


Data sources you can use with the Exchange 2003 module

The data sources you can use for the Exchange 2003 module’s configuration are:

◆ SRDF Synchronous (SRDF/S)

◆ SRDF Asynchronous (SRDF/A)

◆ MirrorView

◆ EMC Mirroring for Windows

◆ Shared Disk Device for Windows

◆ Composite data source

When the Exchange 2003 module is installed with the SRDF data source or the MirrorView data source, the Exchange 2003 module gives you multinode support and failover to a remote location. A shared disk or a mirror can provide access to the MS Exchange data from all nodes.

As with any data source that you want AutoStart to manage, always consult the data source’s best practices documentation in addition to its AutoStart documentation. For the MirrorView data source, consult the CLARiiON MirrorView best practices documentation; for SRDF, consult EMC SRDF best practices documentation.


Modules

Server configurations and data access with the Exchange 2003 module

Note the following server considerations for use with the Exchange 2003 module:

◆ All MS Exchange servers monitored by the Exchange 2003 resource group belong to the same AutoStart domain and Windows Active Directory domain.

◆ The MSExchange servers can be separated by a WAN and on different subnets.

◆ The Exchange 2003 module is compatible with CLARiiON and Symmetrix.

◆ Dynamic disks are not supported as a Shared Disk Device for Windows data source.

◆ Dynamic disks are supported as an EMC Mirroring data source.

When setting up your MS Exchange environment, consider how all nodes access the common set of MS Exchange data files (databases and transaction log files). There are a few way to provide shared access to the data—for example:

◆ If you are using a shared disk, a single physical disk is accessible to all nodes; AutoStart ensures only one node has access to the disk at a time.

◆ If you are using EMC Mirroring for Windows, the mirroring driver ensures each write operation written to the disk of the active MS Exchange server is replicated to the secondary nodes automatically and synchronously.

The basic approach is the same regardless of the disk configuration. The aim is to split the MS Exchange Server 2003 installation into two parts by putting MS Exchange Server’s:

◆ Programs onto each node’s local hard disk

◆ Databases and transaction log files onto the shared or mirrored disk

Sample configurations for some of these data sources are provided in the following topics:

◆ EMC Mirroring for Windows' use with the Exchange 2003 module

◆ Shared disk device for Windows' use with the Exchange 2003 module

◆ EMC SRDF Mirroring's use with the Exchange 2003 module

EMC Mirroring for Windows' use with the Exchange 2003 module

With the EMC Mirroring for Windows data source configuration (like the one shown in Figure 3), a module instance of MS Exchange 2003 runs on a primary (active) machine. A secondary (passive) instance, on a second machine, is idle until failover occurs. A


Modules

dedicated TCP/IP link is used for mirroring. Refer to the AutoStart documentation for configuring this dedicated link correctly. For more information, see “EMC Mirroring for Windows data source” on page 71.

Figure 3 Exchange 2003 using EMC Mirroring for Windows

The EMC Mirroring for Windows data source disks:

◆ must be similar in size.

◆ must not have an operating system or pagefile.sys file.

◆ must be dedicated solely to MS Exchange data.

Use multiple EMC Mirroring for Windows data sources to accommodate performance or size considerations.

Shared disk device for Windows' use with the Exchange 2003 module

With the Shared Disk Device for Windows data source configuration, the disk is used solely for the MS Exchange 2003 data accessed by the active server and all passive servers, but not simultaneously. MS Exchange Server 2003 program files are installed on the servers’ local system disks and only MS Exchange Data Store files are moved to the shared disk. For more information, see “Shared Disk Device for Windows data sources” on page 86.

In addition, Shared Disk Devices for Windows may be:

◆ On Symmetrix, CLARiiON, or any other storage array

◆ On iSCSI


Modules

A configuration of this data source when used for Exchange 2003 might look like the example in Figure 4:

Figure 4 Exchange 2003 using a shared disk device

Consider the following when using data sources for Shared Disk Devices for Windows:

◆ The shared disk must be present and accessible for all nodes.

◆ MS Exchange Server 2003 requires NTFS volumes. Refer to the Microsoft Exchange Server 2003 documentation for more information.

◆ This disk is mounted with the same drive letter on all nodes.

◆ Create one instance of this data source in AutoStart using the drive letter.

EMC SRDF Mirroring's use with the Exchange 2003 module

Another possible mirroring configuration exists with the purchase of the SRDF license. Figure 5 on page 114 shows three nodes grouped in Site 1. These nodes have a Shared Disk Device for Windows data source to a Symmetrix LUN that is an R1 device. In this example, R1 is zoned to the three hosts in Site 1. The two hosts grouped under Site 2 have a Shared Disk Device for Windows data source on a Symmetrix R2 device. Lastly, because the two sites are separated by a WAN, the resource group has a node group separator between the hosts in Site 1 and Site 2.

The number of servers in each site depends on your needs.

RequirementsTo use SRDF data sources on Windows, configure a Shared Disk Device for Windows data source. SRDF provides data access on the R1 or R2 side. However, the Windows Volume manager must assign a drive letter to the disk on each node manually through the Disk Management snap-in. Verify your SymmAPI version’s compatibility with the Symmetrix you intend to use.


Modules

Note: You can also use VERITAS Volume Manager to assign drive letters.

Figure 5 Exchange 2003 using SRDF Mirroring

Specifically, consider the following when using data sources for mirrored disks (SRDF):

◆ The first mirror (R1) of the disk must be present and accessible to at least one node.

◆ The second mirror (R2) of the disk must be present and accessible to the remaining nodes in the AutoStart domain.

◆ Put NTFS on the disk’s first mirror and make sure it is recognized on all nodes.

◆ All nodes (R1, R2) must use the same drive letter.

◆ In AutoStart, configure an AutoStart SRDF data source for all of the R1 and R2 hosts.

◆ Configure an AutoStart Shared Disk data source to coordinate access to the disk between node groups:

• The shared disk must be present and accessible for all nodes.

• MS Exchange requires NTFS volumes. Refer to the Microsoft Exchange documentation for more information.

• This disk is mounted with the same drive letter on all nodes.

• Create one instance of this data source in AutoStart using the drive letter.

◆ Install Solutions Enabler Revision 6.2 or above on all nodes for this SRDF configuration.

◆ Create SRDF device groups using the tools provided in the Solutions Enabler (for example, SymCLI).

For more information on SRDF configurations, refer to “EMC SRDF Mirroring data source” on page 76.


Modules

Microsoft IIS module


So that you can make sure your web site or FTP site running with Microsoft Internet Information Server (IIS) remains available, AutoStart has a wizard you can use to easily configure a resource group that consists of your AutoStart domain’s IIS sites.

After you use the wizard to create a resource group of IIS sites, AutoStart manages the resource group in the same way it would any other resource group. If a managed IIS site becomes unavailable, AutoStart will try to restart the IIS site’s resource group on the same node; but if it can’t restart the resource group on the same node, AutoStart will relocate the resource group to another node and restart it there. Once you have created the IIS-site resource group, you can manage the resource group as described in “Resource groups” on page 94.

Requirements

If you have web sites or FTP sites running with Microsoft Internet Information Server (IIS), AutoStart provides a wizard that lets you easily configure your sites into one resource group so that you can easily manage them and they remain available. AutoStart supports both WWW (w3svc) and FTP (msftpsvc) site types. Each target server will contain an identical configuration as stored in the IIS metabase.

Before you can use the AutoStart wizard for your IIS sites, your system must meet the following minimum requirements:

◆ Microsoft Windows 2003 installed on all target servers.

◆ Microsoft IIS 4.0, 5.0, or 6.0.

◆ Microsoft .NET Framework 1.1 and 2.0.

◆ A licensed copy of EMC AutoStart.

Note: Before you proceed, consult the latest revision of the EMC AutoStart Release Notes for the version of AutoStart that you are running. The Release Notes may contain additional requirements or restrictions. Refer to the most-recent EMC AutoStart Compatibility Guide for the versions that are supported for use with AutoStart. For help obtaining any EMC documentation, see the “Preface” to this guide.

Before you begin

You must complete the following setup before you can use the wizard to create a resource group that monitors your Microsoft IIS sites:

◆ Common user accounts (Windows domain or workgroup accounts) must be created for all site operators and anonymous users. Local accounts will not be recognized upon failover.

◆ AutoStart must be installed, configured, and running on all target nodes.

◆ The nodes in your AutoStart domain and IIS must be installed and configured before you can create the module for IIS.

Microsoft IIS module 115

Modules

◆ All managed IP addresses and data sources that are to be used with IIS must be configured before you configure the AutoStart module for IIS.

Windows Print Services module


Using the Microsoft Windows Print Services (WPS) module on two Windows servers lets you reduce downtime due to the failure of a machine, application, or disk. You can start, stop, and monitor Windows Print Services as well as provide failover and availability tracking for Windows Print Services. The module consists of a wizard that creates a resource group you can use just like any other resource group in AutoStart.

Once you set up a resource group for Windows Print Services and bring it online, WPS services are constantly monitored by AutoStart. The module continually monitors both the Print Services and the node upon which these services are running. If the print spooler should fail, AutoStart attempts to restart it. If the print spooler cannot be restarted, AutoStart relocates the print spooler to the other node. If print services relocate to another node while the printer is working on a print job, the entire print job will be sent to the printer again along with any other print jobs that are in the queue.


Information for a Windows Print Services (WPS) resource group or module instance is provided in the following tabs in the AutoStart console:

◆ Resource groups — If you are using the console as an advanced user, you can view the following WPS resource group tabs: Settings tab, Options tab, Advanced tab, Availability Tracking tab, and Status tab. You can see a list of all resource groups in the domain by viewing the resource group table.

◆ Module instances — Whether or not you are using the console as an advanced user, you can view the following WPS module instance tabs: Settings tab, Resources tab, Notification tab, Status tab, and Availability tab. You can see a list of all instances of Windows Print Services by viewing the Windows Print Services Summary tab. You can see license information and module nodes by viewing the Windows Print Services Configuration tab.


Failover configurations for Windows Print Services

Either a shared disk or a mirrored disk can be used to provide access to the print spooler directory from both nodes. This module only supports active-passive configurations (like the one shown in Figure 6 on page 117), which means that the Windows Print Services can run on only one node at a time.


Modules

The module for Windows Print Services provides high availability for network clients printing to network print devices. Network client computers print to the network printer through an alias name and a printer share. The alias name and printer share are moved to the server that is currently online. The print spooler cached data is stored either to a shared disk or a mirrored volume.

With the shared disk configuration using the Shared Disk for Windows data source (as shown below), the shared disk is used solely for the Print Spooler cached data that will be accessed by both the active server and the passive server (but not at the same time). The Windows Print Spooler is installed on each server.

Figure 6 WPS with shared disk configuration

Windows Print Services module 117

Modules

If the print spooler cached data is stored to a mirrored volume using the EMC Mirroring for Windows data source (as shown Figure 7), the mirrored partitions must be of similar size, must not have an operating system or pagefile.sys file on them, and should be dedicated solely to Print Spooler cached data.

Figure 7 WPS with cached data stored to a mirrored volume

Oracle module for Windows

Note: This feature is available on Windows.

The Oracle AutoStart module provides high availability for network clients accessing databases. Network client computers connect to the database through a virtual IP address. The IP address is moved to the server that is currently online. The database data files are stored on a data source.

Either a shared disk or a mirrored disk provides access to the database in a two-node domain. This module only supports active/passive configurations, which means that the database can run on only one node at a time. The AutoStart module for Oracle on Windows supports the following data sources:

◆ EMC Mirroring for Windows data source for use with Oracle on Windows

◆ Shared Disk Device for Windows data source for use with the Oracle module

◆ SRDF Synchronous (SRDF/S) and SRDF Asynchronous (SRDF/A)

The AutoStart module for Oracle on Windows, which must be used in conjunction with AutoStart, starts, stops, and monitors Oracle 9i and Oracle 10g databases for Windows and the node on which they run, and provides availability tracking for the database, and either restarts or moves the database in the event of failure.

EMC Mirroring for Windows data source for use with Oracle on Windows

When you use the EMC Mirroring for Windows data source, data written to the active server is mirrored to the passive server. Mirroring traffic between the two servers travels across a separate, dedicated data link. For more about EMC Mirroring and the AutoStart


Modules

data source that manages it, see “EMC Mirroring for Windows data source” on page 71.

When an EMC Mirroring for Windows data source is used, the following is assumed:

◆ The two Windows-based servers are similar in RAM, processor speed, and disk technology.

◆ Both nodes of the mirror use the same drive letter.

◆ A dedicated data link has been set up.

With the EMC Mirroring for Windows data source configuration shown in Figure 8, an instance of Oracle runs on a primary (active) computer. A secondary (passive) instance is idle until failover occurs. A dedicated TCP/IP link is used for mirroring.

Figure 8 Oracle configuration with EMC Mirroring on Windows

Shared Disk Device for Windows data source for use with the Oracle module

When you use the Shared Disk Device for Windows data source like the one shown in Figure 9, two or more nodes in the AutoStart domain are physically connected to a single shared disk device using connections. The Shared Disk Device for Windows data source allows AutoStart to determine which node has access to the disk to perform read and write operations.

Figure 9 Oracle configuration with a Windows shared disk

Oracle module for Windows 119

Modules

Shares that are defined before you define the Shared Disk Device for Windows data source are captured when you attach the data source for the first time. Once the data source is attached, the AutoStart agent monitors for new and deleted shares. For more about the AutoStart data source that manages shared disk devices, see “Shared Disk Device for Windows data sources” on page 86.

When a Shared Disk data source is used, make sure:

◆ The disk connections have been set up correctly.

◆ All connected nodes are online and recognize the disk.

◆ The drive has been defined in AutoStart as a shared disk.

EMC requires that you use redundant networks for shared disks. If network partition occurs between the machines accessing the disk, communication is lost and there are no checks to ensure that only one machine accesses the disk at a time. Disk corruption may occur due to concurrent writes.

EMC recommends that Windows shared disk configurations use like controllers. Using different controllers may cause unexpected behavior.

With the shared disk configuration, the shared disk is used solely for the database data that is accessed by both the active server and the passive server, but not at the same time. Oracle is installed on each server.

EMC SRDF Mirroring data source for use with the Oracle module on Windows

The SRDF data source supports AutoStart modules, such as Oracle. In a typical configuration (like the one in Figure 10 on page 121), the SRDF data source is a resource verified prior to allowing dependent resources such as file systems, volume groups, and databases to come online. The SRDF data source monitors SRDF links and device states to ensure the remote replication is available. SRDF provides a vital link, allowing you to


Modules

manage failover and failback control operations at the host level. For more about the AutoStart data source that manages SRDF, see “EMC SRDF Mirroring data source” on page 76.

Figure 10 Multinode Oracle configuration with SRDF

The SRDF data source requires the following:

◆ First mirror (R1) of the disk is present and accessible to at least one node.

◆ Second mirror (R2) of the disk is present and accessible to the remaining nodes in the cluster.

◆ NTFS on the disk’s first mirror and recognized on all nodes.

◆ All nodes (R1, R2) use the same drive letter.

◆ An AutoStart SRDF data source is configured for all of the R1 and R2 hosts.

◆ An AutoStart Shared Disk Device for Windows data source coordinates access to the disk between node groups.

◆ Solutions Enabler Revision 6.2 or later is installed on all nodes for this SRDF configuration.

◆ SRDF device groups are created using the tools provided in the Solutions Enabler (for example, SymCLI).

◆ A dedicated link is required.

An additional mirroring configuration exists with AutoStart’s EMC SRDF data source. Figure 10 shows three nodes grouped in Site 1. These nodes have a Shared Disk Device for Windows data source connected to a Symmetrix LUN that is an R1 device. In this example, R1 is zoned to the three hosts in Site 1. The two hosts grouped under Site 2 have a Shared Disk Device for Windows data source on a Symmetrix R2 device. The resource group has a node group separator between the hosts in Site 1 and Site 2.

Oracle module for Windows 121

Modules

For more information about using SRDF with Oracle, refer to the Using EMC SRDF Family Products with Oracle RDBMS engineering white paper.


Information for a Windows Oracle resource group or module instance is provided in the following tabs in the AutoStart console:


◆ Module instances — Whether or not you are using the console as an advanced user, you can view the following Oracle module instance tabs: Settings tab, Resources tab, Notification tab, Status tab, and Availability tab. You can see a list of all instances by viewing the Oracle on Windows Summary tab. You can see license information and module nodes by viewing the Oracle on Windows Configuration tab.


Oracle module for UNIX

Note: This feature is available on UNIX.

Use the AutoStart Oracle module on UNIX to create module instances in AutoStart that behave like any resource group in AutoStart. An Oracle module instance starts, stops, and monitors Oracle databases for UNIX, and provides availability tracking for the database.

Use a shared disk to provide access to the database data from both nodes. This module only supports active/passive configurations, which means that the database can only run on one node at a time. See “Failover configurations for Oracle modules on UNIX” on page 123 for a list of ways you can use AutoStart to manage Oracle databases on UNIX. The AutoStart module for Oracle is supported on multiple nodes and works with SRDF and MirrorView data sources.


Information for a UNIX platform’s Oracle resource group or module instance is provided in the following tabs in the AutoStart console:


◆ Module instances — Whether or not you are using the console as an advanced user, you can view the following Oracle module instance tabs: Settings tab, Resources tab, Notification tab, Status tab, and Availability tab. You can see a list of all instances by viewing the Oracle on UNIX Summary tab. You can see license information and module nodes by viewing the Oracle on UNIX Configuration tab.


Modules


Failover configurations for Oracle modules on UNIX

The AutoStart module for Oracle provides high availability for network clients accessing databases. Network client computers connect to the database using a virtual IP address. The IP address is moved to the server that is currently online. The database data files are stored to a data source, such as the:

◆ UNIX File System type data sources, such as Solaris File Systems, HP-UX File Systems, and AIX File Systems (refer to “UNIX file system type data sources” on page 92)

◆ The EMC SRDF Mirroring data source (refer to “EMC SRDF Mirroring data source” on page 76)

Shared diskWith the shared disk configuration (see Figure 11 on page 123), the database files are stored to a shared disk, as shown in the following active/passive configuration.

Note: EMC recommends that you use redundant networks when using shared disks. If network partition occurs between the computers accessing the disk, communication is lost and there are no checks to ensure that only one machine accesses the disk at a time. Disk corruption may occur due to concurrent writes.

Figure 11 Shared disk Oracle configuration on UNIX

The shared disk is used solely for the database data that is accessed by both the active server and the passive server, but not at the same time. The same version of Oracle is installed on each server, and each server is running the same operating system (for example, all Solaris, all AIX, or all HP-UX).

SRDFAnother possible mirroring configuration exists with the purchase of the SRDF license. The SRDF data source provides application availability to critical data, even in the event of node, network, or resource failure. Figure 12 shows three nodes grouped in Site 1. These

Oracle module for UNIX 123

Modules

nodes have a shared disk data source connected to a Symmetrix LUN that is an R1 device. In this example, R1 is zoned to the three hosts in Site 1. The two hosts grouped under Site 2 have a shared disk data source on a Symmetrix R2 device. The resource group has a node group separator between the hosts in Site 1 and Site 2.

Figure 12 SRDF Oracle configuration on UNIX

For more information about managing an SRDF data source with AutoStart, refer to “EMC SRDF Mirroring data source” on page 76. For more information about using SRDF with Oracle, refer to the Using EMC SRDF Family Products with Oracle 9i RDBMS engineering white paper.


CHAPTER 7Processes and Services

This chapter describes processes and services that AutoStart can monitor:

◆ Processes, services, and proxies ........................................................................... 126◆ Processes ............................................................................................................. 127◆ Process proxies..................................................................................................... 129◆ Configurations for processes and process proxies ................................................. 130◆ Services................................................................................................................ 131◆ Utility processes ................................................................................................... 132

Processes and Services 125

Processes and Services

Processes, services, and proxiesAn AutoStart domain can contain these types of processes:

◆ A standard process is a user process. UNIX standard processes are run under any user ID. Windows standard processes are not Windows -based services. They must run under the same account as the AutoStart agent.

◆ A service is a Windows-based service.

◆ A utility process is a temporary process that can be started by AutoStart. AutoStart does not monitor the execution of a utility process.

An AutoStart-managed process is a standard process or a service that is defined to and started by an AutoStart agent. An AutoStart agent can start, stop, monitor, relocate, and restart a managed process. Managed processes fall into the following categories:

◆ An aware process is a standard process or a service that is linked with and uses the AutoStart runtime library of functions. An aware process can publish its own sensors and actuators. For more information, see Chapter 8, “Rules, Triggers, and Programming Tools.”

◆ An unaware process is a standard process or service that is unaware of AutoStart. These processes include third-party and shrink-wrapped software. An unaware process cannot publish its own sensors and actuators.

Before AutoStart can manage and monitor a process or service, you must configure the process or service in AutoStart in a way that defines the characteristics of the process or service. For example, in order for AutoStart to manage a service, you must create an AutoStart service that maps the physical service name to a logical AutoStart name that AutoStart can use to relocate the service from node to node. Each managed process or service has one built-in sensor called ft_ProcState. You can use this sensor to monitor the state of the service or process.

Proxy processes: Node proxies and process proxies

For additional sensors and management capabilities, use proxy processes. Proxy processes are special processes that have a link in the AutoStart programming library and therefore can publish additional sensors to the environment. AutoStart supports two types of proxies: node proxies and process proxies:

◆ A process proxy provides sensors on behalf of another process, which does not have the AutoStart programming library built in. For more information, see “Process proxies” on page 129.

◆ Node proxies provide sensors for the operating system. For more information, see “Node proxies” on page 52.

AutoStart provides a default node and process proxy as a standard part of the environment. You can define additional sensors and proxies for monitoring additional data points.



Process definitions

To manage a nonservice process, you create an AutoStart process definition. A process definition defines the start, stop, and test procedure for a process as well as the user name, environment variable, and working directory under which the process is to be run. Process definitions are used primarily on UNIX nodes, but can be defined on Windows nodes.

Service and process failure

Processes and services are managed by the AutoStart agent. When AutoStart starts a process, the agent monitors the application. If the application fails while it is being monitored, the agent detects its state change and updates its AutoStart state. If it is in a resource group it will typically be restarted or relocated.

The actions that the agent takes upon application failure can be configured. If the process or service is running stand-alone (single node configuration), the Auto Restart option is chosen and the process or service will be restarted.

If the process or service is included in a resource group, the agent initiates the procedure to restart the application based on the restart option selected for the resource group. If the restart on the current node is unsuccessful, the resource group relocates to another node in the preferred node list.

ProcessesUNIX daemon processes can be managed and monitored by AutoStart. However, before processes can be managed, they first must be defined or described to AutoStart. Once the process is defined, it can be started, stopped, monitored, and managed by AutoStart. When put in a resource group, processes can be restarted on failure or relocated to another node for maintenance or failure.

The information AutoStart needs to manage a process is stored in a configuration. The idea is to create a configuration for each node that the process runs on. You aren't, however, limited to one node per configuration; you can create a configuration that covers multiple nodes. You'll have to decide whether you can combine nodes on a configuration for the process, then create configurations for the process so that it can run where it needs to run.

Processes can be managed within an AutoStart domain by creating a definition for the process. Whenever the software performs an operation involving that process, it uses this definition to access the application. When defining a process, define a configuration for that particular process, which allows for multiple processes defined differently for different nodes.

By specifying the node or nodes where the configuration is valid, the software automatically chooses which configuration to use when the process starts. If the configuration is the same for all nodes in the AutoStart domain, select the All Nodes option. Once configured, the process can be managed from a resource group, or stand-alone.

The same process can have multiple configurations. These different configurations enable the process to run on various nodes where the conditions needed to run the process may be different. For example, the process may be installed in C:\directory\process.exe

Processes 127


on node gold, and D:\directory\process.exe on node silver. Separate configurations for each of these nodes ensures that the process runs correctly on each node. Create only one configuration per node.

Note: Do not create multiple configurations for the same node for the same process. If one node has multiple configurations defined for it, the configuration that AutoStart chooses when the process starts cannot be predicted.

Note: If there is a node-specific configuration and an All Nodes configuration, the node-specific configuration takes precedence.

Once configured, the process can be placed in a resource group, or managed stand-alone. When a process is included in a resource group, the process must be installed on all nodes in the resource group’s preferred node list and must be able to run correctly on those nodes.

For processes that run on only one node, but need to be restarted on failure, AutoStart provides two restart options: Restart on Failure and Restart on No Response. These options must be turned off when a process is being managed by a resource group.

If the process is included in a resource group, it is monitored and managed, ensuring that the process remains running. In the event that the process cannot run on that node, the software ensures relocation to and operation on another node in the cluster. When a process is included within a resource group, ensure that it is installed on each node in the cluster and can run correctly on those nodes.

Processes can also be monitored and managed from rules using the rule API calls. For more information on rule API refer to “Rules” on page 134 and “AutoStart's API subroutines” on page 160

Before you begin

Before a process can be managed, it must be configured. To configure a process you must:

◆ Name the process.

◆ Define the start and stop operations.

◆ Select a class name for publishing sensors.

◆ Select a proxy process for the process (optional).

◆ Configure startup parameters.

◆ Determine the process’s actions in the event of existence or response monitor failure.

Process tabs in the AutoStart console

A process’s AutoStart information is provided in the following tabs in the AutoStart console:

◆ The Settings tab

◆ The Sensors tab

◆ The Actuators tab



A process’s configuration’s information is provided in the following tabs in the AutoStart console:

◆ The configuration’s Settings tab

◆ The Options tab

◆ The Start Script tab, and the Stop Script tab

You can see a list of all of an AutoStart domain’s processes by viewing the process table.

Process proxiesA process proxy is an AutoStart-aware process that you create so that it can publish sensor values and accept actuator function calls on behalf of another process or proxy. Proxies provide aware proxy functionality to unaware processes. When you associate a proxy with a process, it starts and stops with the process it is managing.

Proxies are useful for adding additional management capabilities through sensors and actuators to applications that have a command line interface or a published API. The proxy process gathers sensor values from the unaware application and reports them to the AutoStart agent. When the agent fires an actuator, the proxy runs the function that can affect the unaware process.

For example, a process proxy for a database may publish a sensor that reports memory usage or transaction rates on behalf of the managed database process. For more information, see “Sensors” on page 147.

Proxies can also define actuators (control functions) that take action or modify an application’s environment. For example, a database proxy could have an actuator method that, when called, truncates the transaction log files. AutoStart lets you associate a proxy process with the managed database process so they can be managed, started, and stopped as a single entity.

You configure a process proxy in much the same way you configure a standard process. As with a process, you can create multiple configurations for a process proxy, which lets you define different operation properties for individual nodes. The idea is to create a configuration for each node that the process proxy runs on. You aren't, however, limited to one node per configuration; you can create a configuration that covers multiple nodes. You'll have to decide whether you can combine nodes on a configuration for the process proxy, then create configurations for the process proxy so that it can run where it needs to run.

Process proxy tabs in the AutoStart console

A process proxy’s AutoStart information is provided in its Settings tab in the AutoStart console. A process proxy’s configuration’s information is provided in the following tabs in the AutoStart console: the configuration's Settings tab, the Options tab, the Start Script tab, and the Stop Script tab. You can see a list of all of an AutoStart domain's process proxies by viewing the process proxy table.

Process proxies 129


Configurations for processes and process proxiesA configuration specifies the information needed for AutoStart to start and stop a process or proxy process on a node. Using a configuration, a single process or proxy process can run on the node or set of nodes using specific input factors for that node or set of nodes.

Typically, the input factors are identical for all nodes where a process might run, which means you need only one configuration for the process. However, if the process's input factors differ by node, you will need to set up multiple configurations for the process.

What is on the configuration?

A configuration identifies the following input factors. If any of this information differs depending on the node where its process or proxy process will run, you will need to create multiple configurations.

◆ Startup executable, script, or command, with startup parameters

◆ Shutdown executable, script, or command, with shutdown parameters

◆ Login account information

◆ The input/output directory or directories

◆ The response monitor

◆ The existence monitor

◆ Restart options

◆ Environment variables

For a single named process or proxy process, you generally do the following:

◆ Create one configuration for all nodes (called an all-node configuration). An all-node configuration is a configuration in which the All Nodes option is chosen on the configuration’s Settings tab.

◆ Then, create an overriding, node-specific configuration for any node that requires different input factors. A node-specific configuration is a configuration in which the Selected List option is chosen on the configuration's Settings tab. Note that you can specify more than one node in a node-specific configuration; it is not required that you limit it to one node.

The steps for defining configurations for both processes and proxy processes are the same. Furthermore, the name you give to a configuration can be significantly important, especially if you are creating more than one configuration for a process or proxy process; for guidelines on naming configurations and other AutoStart resources, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide.

Note: In general, it is recommended that you avoid creating multiple configurations for the same node for a process or proxy process.

Tabs for a configuration

Configuration information is provided in the following tabs in the AutoStart console:



◆ The configuration’s Settings tab

◆ The configuration’s Options tab

◆ The configuration’s Start Script tab, and the Stop Script tab

Troubleshooting configurations

It is important that you create the fewest number of configurations possible, and name them appropriately for their use. How you set them up and how you name them can affect how AutoStart uses them. For example, create one configuration called Default that lists all nodes, and then create one configuration for node A, which has a unique input and output directory. If you intend to create a configuration structure that is more complicated than this example, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for important information about configuration names.

If a process can not be started from a node using the AutoStart console, you should try launching the process manually from the node’s command line prompt and see if it returns any error. The resulting output would help you in diagnosing the problem.

ServicesA Windows service is a process that performs a specific system function and often provides an API for other processes to call. Windows services and UNIX daemon processes can be managed and monitored by AutoStart. However, before AutoStart can manage a service, you must first define it and describe it to AutoStart.

Once you define the service as a managed service in AutoStart, AutoStart can start, stop, monitor, and manage it as a standalone service or as part of a resource group that manages an application:

◆ If it is a standalone service that runs on only one node, AutoStart can start, stop, monitor, and manage it. You simply define the service to AutoStart.

◆ If the service is part of an application, it must be included in the resource group that manages the application. You must add it to the resource group and manage it using the options available through the resource group. When you include the service in a resource group, services can be restarted on failure, relocated to another node for maintenance or when there is a failure.

If you need advanced monitoring and management, services can be automated via a rule written using the AutoStart rule API calls.

Prerequisites for services

Services that AutoStart manages must meet these requirements:

◆ Before you can include a service in a resource group, the service must be installed on all nodes listed in the resource group’s preferred nodes list, and the service must be able to run correctly on those nodes.

◆ If AutoStart is managing a service, you must make sure you use the MS Windows Service Manager to configure it to Manual start. AutoStart cannot manage a service that is configured to be automatically started on system startup.

Services 131


Service tabs in the AutoStart console

A service’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab, the Sensors tab, the Actuators tab. You can see a list of all of an AutoStart domain’s services by viewing the service table.

Utility processes Typical utility processes are created in order to perform short-lived maintenance activities, such as cleaning up disk space or changing a configuration. There are two types of utility processes:

◆ A standard utility process can be an executable, Perl script, UNIX shell script, or batch file that is launched as part of a resource group, from the AutoStart console, or by a rule. There is no difference between a standard utility process and any other process you define in AutoStart, except that the standard utility process is typically used exclusively for maintenance activities and AutoStart does not monitor it once it starts.

◆ A Telnet utility process runs a Telnet session from the AutoStart console to a remote node in another location. Telnet is the standard Internet protocol for accessing a remote server, and with AutoStart it is typically used for accessing a remote UNIX box. This utility process (UtilProc) runs from any node, and can be launched from the AutoStart console, the command-line interface (CLI), or from a Perl shell script.

Once a utility process is launched, AutoStart does not manage it. The utility process runs but is not monitored or managed in any way other than to provide for the option of waiting for completion. Utility processes are often used to perform some type of maintenance such as cleaning up disk space or running a script, and, therefore, typically do not require monitoring.

Utility process tabs in the AutoStart console

A utility process’s AutoStart information is provided in the following tabs in the AutoStart console: the standard utility process Settings tab (or the Telnet utility process Settings tab), the Options tab, and the Script tab. You can see a list of all utility processes in the AutoStart domain by viewing the utility process table.


CHAPTER 8Rules, Triggers, and Programming Tools

This chapter describes how you can use rules and the AutoStart software developer’s kit to customize resource groups:

◆ Rules .................................................................................................................... 134◆ Triggers................................................................................................................. 139◆ Sensors ................................................................................................................ 147◆ Actuators .............................................................................................................. 149◆ Planning a rule...................................................................................................... 150◆ Guidelines for writing rule text .............................................................................. 155◆ AutoStart's API subroutines .................................................................................. 160◆ Use APIs to make processes AutoStart-aware ........................................................ 164◆ AutoStart's global variables .................................................................................. 166

Rules, Triggers, and Programming Tools 133

Rules, Triggers, and Programming Tools

RulesA rule is a Perl script that is an instruction that runs on an AutoStart domain and defines what would happen to resource group when a certain event is triggered. Rules perform specialized actions in response to events an AutoStart domain, and they act only on a resource group (not individual resources in the resource group). Because AutoStart knows where the resources in a resource group are hosted, the rule acts on the nodes where the resource group is being hosted.

Why use a rule?

By creating and using rules, you can enhance the performance of an AutoStart domain's resource groups, or of any services or processes that you may be managing from the AutoStart console. For example, a rule can:

◆ Start a service or process.

◆ Assign IP addresses.

◆ Bring resource groups online.

◆ Run a script.

Even though each resource group provides its own, well-defined set of resource monitoring and relocation capabilities, you can create rules that enhance what the resource group can do. You can create rules that:

◆ Provide additional procedures in the AutoStart domain while monitoring and managing resources in the domain like resource groups do.

◆ Manage a resource group dynamically. A resource group that isn’t managed by a rule gets relocated to a node in its preferred nodes list; this availability decision is static and made in advance. But when you create a rule that manages the resource group, you can handle availability decisions more dynamically, and have more control in responding to changes in the environment based on up-to-the-second runtime inputs.

◆ Manage a nodeless resource group. In fact, a nodeless resource group must be managed by a rule, because a resource group that has no preferred nodes has no way of knowing which node to relocate to. (For more information, see “Nodeless resource groups” on page 95.)

◆ React to dynamic conditions. Rules can respond to domain-wide information, and can react to dynamic changes in the environment using up-to-the-second runtime information. For example, you can create a rule that monitors CPU usage and relocates a resource group to another node if usage goes above a threshold, such as 90%.

Some basics about rules

Part of understanding what rules can do means understanding their abilities and limitations:

◆ When a rule takes action, it acts on a resource group; that is, a rule cannot take action on an individual resource in a resource group.



◆ Each rule has visibility to, influence over, and control over the entire AutoStart domain; you do not have to duplicate it or customize it for each node. One rule can handle events on all nodes in the AutoStart domain.

◆ The rule is run by an interpreter, which you can read more about in “The rule interpreter” on page 138.

◆ Because a rule will run on any primary agent in the domain, you must make sure that you write your rules so that they are node-independent. When a trigger fires, a trigger event message is sent to the appropriate rule interpreters running the rules associated with the trigger. That rule interpreter then evaluates the rule or rules and initiates the actions specified in the rule(s).

How to create a rule:

1. Plan the rule.

2. Identify the sensors you'll need to use. Define any sensors you need that AutoStart does not supply.

3. Create a trigger definition in AutoStart for each trigger that will drive the rule. Note that you can reuse any triggers that are already defined in the AutoStart domain.

4. Define any actuators that the rule will need.

5. Write the rule text.

6. Debug the rule.

7. Create a rule definition in AutoStart.

8. When you are ready to use the rule, enable it.

What makes up a rule?

In addition to the Perl script that comprises the rule text, AutoStart rules are based on a system of triggers, sensors, and actuators, that provide the foundation of its management and monitoring capabilities. This combination of sensors, actuators, and triggers provides a powerful tool kit for solving a wide range of availability and clustering problems:

◆ A trigger is the defined condition that determines when a rule is executed. A trigger fires on a schedule, on demand, or as a result of a sensor value or event. For more information, see “Triggers” on page 139.

◆ Sensors provide data values to triggers for managed resources such as nodes, processes or other networks. Triggers provide the monitoring and reporting facility on top of the sensors. When the trigger detects a specific condition, it reports the state and sensor value to one or more associated rules. For more information, see “Sensors” on page 147.

◆ When a rule executes, it can perform almost any action from starting a process to attaching a disk to sending a page. If a function that a rule needs to carry out is not directly available in the rule API, rule can alternatively access that function using:

• An actuator

• A native operating system function

• An executable

Rules 135


◆ An actuator is a function call made by an AutoStart-aware process. A rule fires the actuator to make the call; but you can fire it using the AutoStart console, too. Sensors and actuators can exist within the same aware process. An actuator can take an action for a process other than the process whose sensor value caused the trigger to fire. For more information, see “Actuators” on page 149.

◆ The rule text, written in Perl as a script, contains the logic of the rule itself and is what pulls together the system of triggers, sensors, and actuators. You include the rule's Perl script in the rule definition that you create in AutoStart. For more information, see “Guidelines for writing rule text” on page 155 and “Some basics about rules” on page 134.

Resource groups and rules use this system of triggers, sensors, and actuators like this:

◆ In rules, triggers, sensors, actuators, and the rule text are visible to you, and you configure them to your needs.

◆ AutoStart automatically creates them for you when behind-the-scenes when you create a resource group or module instance.

For more information on Actuators, see “Actuators” on page 149

Rule tabs in the AutoStart console

A rule’s AutoStart information is provided in the following tabs in the AutoStart console: the Settings tab and the Rule Script tab. You can see a list of all of an AutoStart domain's rules by viewing the rule table.

How a rule runs

As long as there is one active primary agent running when a rule’s associated trigger fires, a rule gets an opportunity to run. In general, a rule runs when any of the following conditions is met:

◆ A rule driven by at least one sensor-based trigger or one event-log trigger runs as soon as the rule is enabled. (Rules driven only by scheduled or on-demand triggers do not run when the rule is enabled.)

◆ When the condition of a trigger that drives the rule changes state, that is changes from On to Off or from Off to On.

◆ If a rule is driven by more than one trigger, the rule runs each time any one of its triggers fires.

◆ When the process that provides the sensor for a sensor-based trigger stops, the rule runs when the process is restarted.

◆ When the rule relocates to a different rule interpreter because of a failure, the rule runs on the new rule interpreter, unless it is driven only by a scheduled or on-demand trigger.

Multiple triggers for a ruleA rule driven by multiple triggers is evaluated by a rule interpreter when any of the rule’s associated triggers fire. Do not assume when the rule runs that all triggers associated with the rule have fired. If your rule requires that all trigger states are true before an action can take place, you must code the rule for that event using conditional control statements.



If the rule is to perform an action based on the trigger that fired it, you can use AutoStart-supplied global variables to determine which trigger fired, the node on which the trigger fired, and other useful information.

When a rule is enabled all of its associated triggers are activated. However, the rule’s response at the time the rule is enabled differs depending on the type of trigger used to drive the rule:

◆ A sensor-based trigger — Reports an initial value, if possible. For more information, see “Sensor-based triggers” on page 143.

◆ An event-log trigger — Takes no action until the event is reported to the event log. For more information, see “Event-log triggers” on page 141.

◆ A scheduled trigger — Takes no action until the time of the scheduled event. For more information, see “Scheduled triggers” on page 142.

◆ An on-demand trigger — Fires when someone or something specifically invokes it. For more information, see “On-demand triggers” on page 147.

If a combination of trigger types is used and includes a sensor-based trigger, the behavior of the sensor-based trigger takes precedence.

Trigger-related global variables When a trigger fires, before the rule is evaluated, AutoStart creates a set of global variables that the rule can use to obtain information about the trigger that fired and the rule driven by the trigger. This information includes:

◆ The name of the rule

◆ The name of the trigger

◆ For threshold and state sensor triggers, the state of the trigger: ON or OFF

◆ The name of the node executing the sensor monitored by the trigger

◆ Sensor-related information:

• The sensor’s name

• The argument passed to the sensor

• The value and data (if any) that the sensor returns

◆ The condition under which the rule-invoking trigger fired:

$FT::INITIAL_STATE, $FT::CURRENT_STATE, or $FT::NEW_STATE

Some of the trigger-related global variables are associative arrays that are indexed by a trigger name. That is, the associative array lists all trigger names and contains corresponding values for each trigger name, unless explicitly stated in the description. If you try to use, as an index key, a trigger name that has not been chosen to drive the rule or if the rule has not yet received an initial value from a trigger that drives it, the resulting value is undefined.

Consider the following factors when using the trigger-related global variables in a rule driven by multiple triggers:

Rules 137


◆ When a rule is enabled and an initial sensor value for one trigger is received, the rule is executed and the global variables for this one trigger are now defined. The global variables for triggers that have not yet received an initial sensor value are undefined.

◆ When a rule is enabled, it is executed at least once for each trigger that drives it. After one initial sensor value for a trigger is received, the global variables for this trigger are defined.

◆ An AutoStart-aware process of a given class must publish the same set of sensors. If a trigger monitors a sensor that is present in multiple processes, then a trigger event is received for each monitored sensor. The rule runs at least once for each trigger event.

Based on these factors, a rule driven by multiple triggers should be written so it references global variable data associated with a trigger only after the rule is sure that the trigger has received an initial value for the monitored sensor. For example, assume a rule is driven by Trigger1 and Trigger2. The rule is enabled and, subsequently, receives an initial sensor value from Trigger1. When the rule then runs, if it attempts to access an AutoStart global variable associated with Trigger2 (for example, referencing $ft::OriginNode{Trigger2}), the value of this global variable is undefined.

The rule interpreter

A rule interpreter resides on each primary agent node. When a rule is enabled, AutoStart chooses a primary agent on which to run the rule. This can be any primary agent, so for this reason rules must be node independent. AutoStart randomly selects the rule interpreter so that rule work is spread among the active rule interpreters. A single rule interpreter can be responsible for multiple rules.

The AutoStart rule interpreter monitors the AutoStart domain resources and responds as needed. Because the rule interpreter conceptually sits above the AutoStart domain, it has access to all the domain's resources and can respond as needed.

Once a rule is enabled, any condition that fires a trigger that drives the rule causes the rule to run. Once AutoStart determines which agent is responsible for the rule, it loads the rule into the rule interpreter on the node where the agent is running.

The rule remains on the agent's rule interpreter until:

◆ You disable the rule.

◆ The node or rule interpreter fails. If the node or rule interpreter fails, AutoStart immediately relocates the rule to another rule interpreter on a different node. When a rule relocates to a new rule interpreter because of a failover, the rule runs on the new rule interpreter, provided that the condition that fired the trigger is still valid.

One at a timeAny number of rules can be enabled in the AutoStart domain—although a rule interpreter can run only one rule at a time. If you have complex rules that interact with other rules, concurrence may be an issue. You can use the ft::GetRuleLock and ft::ReleaseRuleLock subroutines in such rules, to serialize rule execution so that concurrent access to critical data is not possible.



TriggersA trigger is the object that drives a rule. A rule's triggers identify the conditions under which the rule will execute. When a trigger fires, it forces the rule interpreter (see “The rule interpreter” on page 138) to evaluate the rules that the trigger drives. If the trigger is a sensor-based or event-log trigger, it reports the trigger's state (on or off) and sensor value to the rules that the trigger drives. Criteria that can fire a trigger and make a rule run include:

◆ A custom sensor’s value that crosses a threshold

◆ A node's load that suddenly spikes

◆ An event that is scheduled to occur at a specified time each week

◆ A process failure

◆ A node failure

◆ A process’s load that exceeds a predetermined threshold

Triggers and rules

Any number of triggers can drive a rule, which lets you create a trigger for any condition that makes the rule run. As long as a rule is enabled, its triggers can fire. Also, keep in mind that you can re-use triggers from one rule to another; any trigger can drive more than one rule. If a trigger drives more than one enabled rule, this happens for each rule it drives.

Because triggers make a rule run, it is important that you understand the options available to you for triggers. Defining the correct trigger is the key to developing effective and efficient rules. AutoStart gives you a number of choices. For example, a trigger can:

◆ monitor a single object or a list of like objects.

◆ look for a sensor value that:

• Crosses a threshold condition (such as when disk space usage exceeds 90 percent) or

• Changes in condition (such as available disk space changing by 10 percent).

◆ fire:

• When a sensor reports a specified value has occurred.

• When a specified Windows event occurs.

• At a scheduled time.

• Only on demand.

Triggers 139


When a trigger fires, any number of consequences can unfold, as shown in Figure 1.

Figure 1 Triggers and rules

Types of triggers

You define and configure a trigger and the condition that will make it fire by using the AutoStart console. The console stores triggers in AutoStart’s database. The conditions that make a rule run are defined in the rule’s triggers.

There are four types of triggers:

◆ Sensor-based — Often you want to run a rule when a condition occurs in a process or on a node. In this case, you use a sensor-based trigger. A sensor-based trigger monitors a sensor for a particular condition and fires when that condition is met. A single trigger can monitor no more than one sensor, but may monitor multiple instances of a sensor. AutoStart provides a number of sensors for you to choose from. The proxy processes publish information on behalf of a node or a process, which you can use to trigger a rule. Additionally, you can program sensors. For more information, see “Sensor-based triggers” on page 143.

◆ Scheduled — A scheduled trigger fires once, at a time that you specify; it can fire at a particular time or date, once or on a repeating schedule, such as every Friday at 3:00PM. For more information, see “Scheduled triggers” on page 142.

◆ On-demand — An on-demand trigger fires only when a user or rule intentionally fires it. An on-demand trigger is useful for testing a rule and for sequencing two or more cooperating rules. For more information, see “On-demand triggers” on page 147.

◆ Event-log (available only on Windows) — A Windows event-log trigger watches events that are written to a Windows event log and fires when a specified event is written to the log. For more information, see “Event-log triggers” on page 141.

You choose the type of trigger best suits your needs.

What you should know about using triggers

Here are some factors to consider when using triggers:

◆ All rules associated with a trigger run when the trigger fires.

◆ If a rule is driven by multiple triggers, the rule runs when any of the triggers fires.



◆ A sensor can be monitored by any number of triggers.

◆ A trigger can monitor a node sensor or a process sensor.

◆ A trigger can monitor no more than one sensor. However, the scope of a trigger determines the processes or nodes on which the sensor is running and allows multiple instances of a sensor to be monitored.

◆ It is important to note that a trigger monitors a sensor at the source of the sensor, not over the network from the rule. This localizes the polling and reduces network traffic. In addition, the rate at which a trigger polls a condition is user-definable, allowing you to control the amount of system resources needed for monitoring.

Trigger tabs in the AutoStart console

A trigger’s AutoStart information is provided in its Settings tab in the AutoStart console. You can see a list of all of an AutoStart domain’s triggers by viewing the trigger table.

Event-log triggers

An event-log trigger monitors the events logged to a Microsoft Windows event log.

Windows only. Event-log triggers can be used only on the Windows operating system. Note that this is the same Windows event log that you can view using the MS Windows Event Viewer.

An event-log trigger fires when a Windows event occurs or when a series of Windows events occur within a time interval that you configure. All event-log triggers monitor the event log sensor (ft_EventSensor sensor). Each event log sensor reads a Windows event log and returns the Windows event message that complies with the sensor definition.

Logs that the event-log trigger can pollBecause event-log triggers are tied to event notifications written to the Windows event log, you can configure them to monitor events published to:

◆ A Windows process’s event log (that is, if the process publishes events) or

◆ A node’s Windows event log.

When you enable any rule that monitors the event-log trigger, AutoStart activates the trigger and starts monitoring log file for the event (or events) that will fire the event-log trigger. Windows provides four types of logs; each trigger can monitor one type of log:

◆ Application log

◆ Security log

◆ System or

◆ User-defined log. User-defined logs are created by applications that use the Windows API.

The trigger polls the specified Windows event log at a frequency that you specify. If events that meet the trigger’s conditions are written to the log file that the event-log trigger is monitoring, the trigger fires—that is, it sends a message to an AutoStart agent. When the

Triggers 141


agent receives the message, its rule interpreter executes the rule. This lets you create an event-log trigger that performs an action when a Windows event, or series of Windows events, is logged to a particular type of Windows event log.

Note: It is possible to create an API that will post events, which can then in turn fire event-log triggers. See the ft::PostEvent subroutine for more information.

Event characteristics that can fire an event-log triggerThe following options enable you to identify the event or events that will fire the event-log trigger. You identify the events by specifying any or all of the following characteristics about the events:

◆ The application or system name (for example, the AutoStart Backbone or RepliStor) that generates the event in the Windows event log.

◆ The event ID

◆ The event type (Error, Warning, Information, Audit Success, or Audit Failure)

◆ Text in the event's description

If it takes more than one event to fire the trigger, you can list the events, and specify whether all or only one event must occur. If all must occur, you can also specify:

◆ The sequence in which the events must occur

◆ The length of time over which the events must occur.

Examples An event-log trigger can fire when any of the following events are written to a Windows log:

◆ A warning event or an error event is written to the Windows system log.

◆ A warning event that has the text "fail" in its description followed within 15 seconds by an error event is written to the Windows application log.

◆ An audit failure event is written to the Windows security log.

◆ An event ID of 517 is written to the Windows security log by the source called Security.

Trigger tab in the AutoStart consoleAn event-log trigger's AutoStart information is provided in the Event-log trigger Settings tab in the AutoStart console.

Scheduled triggers

A scheduled trigger fires at a certain time or on a recurring basis. You can define a scheduled trigger to fire at a time and date, just once or on a repeating schedule, such as every Friday at 3:20 PM.

Scheduled triggers don't poll user-defined sensors; instead, they poll a sensor that reports the system time.

Types of scheduled triggersWhen you define a scheduled trigger, you specify the time or the schedule on which you want the trigger to fire. A scheduled trigger can fire at a specific date or time, or can fire at regular intervals. There are two types of scheduled triggers:



◆ Date — When you select this schedule type you can set the trigger to fire at a specific date and time, or to fire on a regular basis, such as every day, every month, or every hour.

◆ Day of week — When you select this schedule type you can set the triggers that can be set to fire on specific days of every week at a set time.

When a scheduled trigger firesAs long as the rule is enabled at the time the trigger is scheduled to fire, the following occurs:

1. A primary agent queries the AutoStart database for the rule definition and its triggers’ definitions; the agent sends the rule text to a rule interpreter (described in “The rule interpreter” on page 138) and then loads the triggers.

2. At the configured time, the trigger fires.

3. The rule interpreter is notified that the trigger’s state is TRUE or ON. If more than one rule uses the trigger, more than one rule interpreter is notified that the trigger fired.

4. The rule interpreter runs the rule body text associated with the trigger.

There are some extenuating circumstances you must take into account when you are coding a rule that will be fired by a scheduled trigger.

Trigger tab in the AutoStart consoleYou configure a scheduled trigger by using the Scheduled trigger Settings tab in the AutoStart console.

Sensor-based triggers

A sensor-based trigger monitor a sensor for a particular condition and fires when the associated sensor value matches the criterion set during trigger definition. A single trigger can monitor no more than one sensor, but may monitor multiple instances of a sensor. In firing, the sensor-based trigger reports the condition to the rule or rules that are monitoring the trigger.

Which sensor?When you're configuring a sensor-based trigger, you must determine if the type of sensor you need is available. If you need to get a sensor value from like processes, you must correctly use AutoStart’s sensor classes. Proper use of sensor classes allows a single trigger to monitor a sensor in a number of like processes.

AutoStart provides a number of sensors (through the AutoStart agent and through proxy processes) for you to choose from. The proxy processes publish information on behalf of a node or a process, and can be used to trigger a rule based on information gained from the node or process. Additionally, you can program sensors. Sensors, either those provided with AutoStart or programmed by you, publish data that can be monitored to drive rules. For more information, see “Sensors” on page 147.

Triggers 143


Rules driven by sensor-based triggersFigure 2 shows how the trigger monitors an AutoStart-aware process; the example shows the rule on a single, primary AutoStart agent in the AutoStart domain. In practice, the rule interpreter runs on additional primary agents in the AutoStart domain, which are sent messages, too. The sequence consists of the following steps:

1. You enable the rule.

2. The primary agent queries the AutoStart database for the rule definition and its associated trigger definitions. The agent sends the rule text to a rule interpreter, then loads the sensor-based triggers into the nodes or processes specified by the trigger’s scope definition. (For more information, see “The rule interpreter” on page 138.) This activates the triggers.

3. Each activated sensor-based trigger immediately sends its current trigger state and the current value of its associated sensor, if any.

4. The rule executes once for each sensor-based trigger; this establishes the rule’s state.

Figure 2 Rules driven by sensor-based triggers

5. Each trigger polls its sensor for the sensor’s current value; polling occurs at the interval you configured in the trigger's definition. A trigger polls its sensor on any process that has the sensor.

If a process that is within the scope of the rule’s trigger starts up at a later time, the process’s sensor is activated upon process start-up and the trigger starts to monitor it at that time. Thus, a rule can be enabled even if a process its triggers monitor is not currently running.

6. If a sensor returns a value that matches the trigger’s condition, the trigger fires—that is, it sends a message to an AutoStart agent.



7. When the agent receives the message, its rule interpreter runs the rule text associated with the trigger. If more than one rule uses the trigger, more than one rule interpreter may receive the firing notification. The rule can then issue a callback to an actuator in the process, which lets the rule control the process's operations or take some other action.

If the rule is enabled, but the process providing sensors is not running, AutoStart still activates the triggers. The rule simply waits because there is no sensor data for it to process. If, at some point the nodeproxy aware process starts and begins providing sensor data, the trigger is immediately loaded into the process, sends out an initial trigger value, and begins monitoring as before.

Here is a tip! If the trigger monitors a state sensor, there is never a delay. Because state sensors are published by the AutoStart agent, and not an aware process, AutoStart does not need to wait for the process to start. As soon as the rule is enabled the initial trigger value for that state is delivered to the rule. Associating a rule with a node state or process state sensor ensures that the rule gets a chance to run as soon as it is enabled, even if the rule ignores the reported trigger value.

Conditions that fire a sensor-based triggerYou'll configure each sensor-based trigger to check the sensor's value at a time interval. At that interval, the trigger checks the current sensor value and compares it to the triggering condition. If the sensor value matches the triggering condition, the trigger fires.

Each trigger can test for one condition only. Which conditions are available for testing depends on whether you are configuring the trigger to check:

◆ A state sensor

◆ Some other sensor

How you configure a trigger for each type of sensor is described next in this topic.

State sensorsYou can configure a sensor-based trigger to check any of the following state sensors supplied by AutoStart: ft_ProcState, ft_NodeState, ft_DataSourceState, ft_IP_AddressState, ft_NicState, and ft_RG_State. State sensors report predefined states. You can configure the trigger to see if:

◆ The sensor returned a specified state for the object

◆ The sensor did not return a specified state for the object

◆ The state changed

For example, you can configure the trigger to check a process's state using the ft_procState sensor, and to fire if, for example, the process’s state:

◆ is Running, or

◆ is not Failed, or

◆ has changed.

This gives you the ability to run a rule under specific conditions regarding the process’s state. By writing a rule that uses such a trigger, you allow the rule to run only when the state is in a condition of your choosing.

Triggers 145


All other sensorsAll other, non-state sensors (that is, sensors provided by AutoStart or user-defined) also report values that the trigger can test for. When the sensor-based trigger uses one of these other sensors, you can configure the trigger to fire under any of the conditions below:

◆ Compare the sensor value to a value that you specify. The trigger can fire when the sensor's value is greater than, greater than or equal to, less than, less than or equal to, equal to, or not equal to a value that you specify when you configure the trigger. This lets you run a rule when the sensor returns a value that meets a specific condition.

The trigger fires when the sensor’s value remains in that condition relative to the value defined for the trigger. The trigger state stays On as long as the condition exists. Once the condition no longer exists, the trigger state shuts Off.

Note: Do not confuse the trigger’s state of On or Off with state sensors. The trigger’s state is always reported regardless of the type of trigger and regardless of the type of sensor it gets information from. Trigger states are described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ Fire the trigger when the sensor’s value changes by a specified amount. The trigger can fire when the sensor's value changes by an amount (that is, a delta) that you specify. The delta is measured in absolute terms. That is, if the delta is 5, the trigger fires when the sensor's value changes by +5 or -5. This lets you run a rule when the trigger sees that a change in the sensor's value has exceeded a specified amount.

The trigger assumes the initial comparison value is 0 (zero), unless you specify a different initial value when you configure the trigger. The trigger fires when the sensor value changes by the specified amount, then the trigger state shuts Off. The value must change by that amount again for the trigger to fire again.

◆ Fire the trigger when the sensor’s value changes by a specified percentage. For this condition, you must set a delta. The trigger fires when the sensor's value changes by the specified amount relative to the last sensor value. This lets you run a rule when the trigger sees that a sensor’s value has changed too radically from one poll to the next. Note that you can specify a percentage of change from 1 to 999 percent.

The trigger compares the sensor’s first returned value to 0 (zero) unless you specify a different initial value when you configure the trigger. For example, you set the delta value at 10 percent. The initial sensor value is 100. If the next sensor value is plus or minus 10 (that is, 90 or lower, or 110 or higher), the trigger fires. Let’s say the sensor value is 90 and the trigger fires.

The next sensor value must change by 10 percent again for the trigger to fire again. Ten percent of 90 is 9, so a sensor value of 81 or lower, or 99 or higher will cause the trigger to fire. So when the next sensor value is 100, the trigger fires again.

◆ Fire the trigger when the sensor's value changes by a specified percentage relative to an amount range. For this condition, you must set a delta and a range. The trigger fires when the sensor value changes by an amount greater than the specified percentage of the range. You can specify a percentage of change from 1 to 999 percent. This lets you run a rule when the trigger sees that a sensor's value has changed too radically from one poll to the next. The trigger compares the sensor’s first returned value to 0 (zero) unless you specify a different initial value when you configure the trigger.



For example, you set the delta value at 10 percent and the range is 0 to 1000, which means that a change of plus or minus 100 in the sensor’s value causes the trigger to fire. You do not provide an initial value of comparison, so it is assumed to be 0. The sensor value returns a value of 90, so the trigger doesn't fire. The next value is 110; the trigger doesn't fire because it changed by 20. The next value is 135; the trigger doesn’t fire because it changed by 25. The next value is 20, so the trigger fires because it changed by 115.

Trigger tab in the AutoStart consoleA sensor-based trigger’s AutoStart information is provided in the Sensor-based trigger Settings tab in the AutoStart console.

On-demand triggers

An on-demand trigger gives you the control to run a rule when you want it to run. An on-demand trigger is not tied to a sensor or a time. Instead, you fire the on-demand trigger from the AutoStart console, the CLI (command line interface), or from a rule.

Only when it is fired does an on-demand trigger deliver values to the rule (or rules) that are driven by it.

You may find on-demand triggers useful in the following circumstances:

◆ Fire an on-demand trigger from the console when you are creating and testing a rule. In this case, firing an on-demand trigger ensures that the rule is running as needed.

◆ Use an on-demand trigger for sequencing two or more cooperating rules.

◆ Use a scheduled trigger to fire an on-demand trigger from a rule.

Trigger tab in the AutoStart consoleAn on-demand trigger’s AutoStart information is provided in the On-demand trigger Settings tab in the AutoStart console.

SensorsA sensor is an AutoStart object that publishes values from a data point such as a counter or environmental state. A sensor provides data about an AutoStart-aware process to the AutoStart runtime, and can be monitored by a sensor-based trigger that, when fired, runs a rule that performs an action in AutoStart. (For more information, see “Sensor-based triggers” on page 143.) It is implemented as a callback function that returns a value when polled by a sensor-based trigger. The sensor’s value is used to evaluate a trigger condition. AutoStart publishes state sensors for managed processes and managed nodes. You can view the sensors that are being published by using the Sensor tab for a running process, service, or node proxy.

During operation, sensors provide data to triggers. Sensor values can also be explicitly polled through:

◆ The AutoStart console

◆ A running rule

◆ The AutoStart command line interface (CLI)

Sensors 147


AutoStart-supplied sensors and user-defined sensors

You can:

◆ Use any of the sensors supplied by AutoStart. As part of its basic package, AutoStart provides basic sensors that return

• The state of processes, nodes, data sources, resource groups, NICs, and IP addresses

• Events as they occur.

These sensors are described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ Define your own sensors. You can create additional sensors that publish data from within an AutoStart-aware process. For more about these sensors, see AutoStart online help and the EMC AutoStart Administrator’s Guide. When you define a sensor, you identify the data that you want the sensor to provide; the resulting sensor can be simple or complex. For example, a sensor can:

• Simply publish the number of clients connected to a server.

• Undertake a more complex task such as examining a long list of resource values and publishing a value that expresses the health of an AutoStart-aware process.

Both types of sensors are monitored at runtime by one or more sensor-based triggers. A rule can access a sensor’s value or data using the %ft::SensorValue and %ft::SensorData variables or the ft::GetSensorValue subroutine. Refer to “How a rule runs” on page 136 for information on the relationship between a sensor and a trigger.

Sensors provided by proxies on behalf of nodes and processes

AutoStart also provides several aware processes, called node proxies, that provide sensors on behalf of nodes and processes:

◆ An integrated node proxy process, called nodeProxy, publishes node sensors, such as load for each node in the domain. By default each node has a node proxy configured for it. The user must start the proxy process to take advantage of its sensors. For more about these sensors, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ A process proxy, called processProxy, publishes sensors on behalf of processes and services. Each managed process or service can optionally have one process proxy associated with it. For more about these sensors, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

You can find the nodeProxy and processProxy programs here:

◆ On Windows, in the $FT_DIR\sdk\proxies\bin directory.

◆ On UNIX, in the $FT_DIR/sdk/proxies/bin directory.

Sensor classes for processes

The sensor class for a process provides the name under which the sensors for that process are published. Using this class name makes it easier to track where the sensors are originating, and can be used with AutoStart’s API process-related subroutines (see



“AutoStart's API subroutines” on page 160).

When a rule monitors one or more processes, it is important that you use the processes’ sensor classes correctly. When you configure a process using the AutoStart console, you define the sensor class under which all sensors associated with the process are published. This sensor class provides a grouping mechanism under which like processes publish a common set of sensors.

ExampleFor example, you can define three instances of Oracle, using processes named Oracle1, Oracle2, and Oracle3. They are all the same application, so you define them all as belonging to the same sensor class, such as Oracle.

You create a rule that uses a sensor-based trigger called TransactionRate. This trigger is one that you created to monitor a sensor that reports the number of transactions per second for the Oracle server. Because all of the instances of the Oracle application are in the same sensor class, you can define a single trigger that monitors the transaction rate at each instance. This is known as defining the scope of a trigger and, when defined correctly, allows a single rule and trigger to be defined to manage multiple instances of a process.

The scope of a sensor-based trigger for a processThe scope of a trigger determines which processes or nodes the trigger monitors. The scope can be defined as a list of specific processes or nodes, or defined as all processes or nodes. When the scope is defined as including all processes or nodes, AutoStart dynamically determines at runtime the list of processes or nodes associated with the trigger.

Although a trigger is only associated with one sensor, it can be configured to monitor many instances of that sensor. Sensors for processes are placed in sensor classes, and many like processes can be a member of a single sensor class. A trigger defined to monitor a sensor in a sensor class can monitor that sensor for multiple processes. For instance, a trigger can be defined to monitor a sensor in ten like processes if all are included in the same sensor class. A single rule driven by such a trigger can handle an event on any of the process or node in the class. If multiple processes or nodes fall within the scope of a trigger, a separate instance of the trigger monitors each instance of the sensor.

If a trigger’s scope includes multiple nodes or processes, the rule writer should be aware of the scope when writing the rule, and design the rule to handle inputs from multiple sources.

ActuatorsAn actuator is a callback function within an AutoStart-aware application that performs an action for the application. A rule can call an actuator to communicate with or control the application process.

Aware processes of the same sensor class must publish the same user-defined actuators. For example, a sensor that publishes the length of a queue might also have an actuator to start another process when the number of entries exceeds a threshold. Actuators can also perform other types of tasks or communications, such as sending a fax or activating a paging device.

Actuators 149


No default actuators are defined for processes that are not AutoStart-aware.

A rule can use the ft::FireActuator subroutine to fire an actuator. The rule can either call the actuator for a specific process or it can call the actuators in all process instances of a specific sensor class. When the rule text invokes the actuator, the actuator’s associated function is executed within the aware application. The application that the actuator takes action upon does not need to be the same process whose sensor fired the trigger.

You can also fire an actuator using the AutoStart console or the AutoStart command line interface (CLI). These methods are useful when you are debugging an actuator. How you do this is described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

The AutoStart software includes the nodeProxy and processProxy programs that contain useful actuators. The nodeProxy and processProxy programs are found in the $FT_DIR\sdk\proxies\bin directory on Windows and $FT_DIR/sdk/proxies/bin on UNIX. For more information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

Planning a ruleA rule defines a management policy, designed to react to the specific needs of your domain. Rules run above the domain, with a domain wide view which allows them to get information and perform actions on a domain-wide basis, not just on a single node. A rule using AutoStart’s subroutine extensions has the power to monitor and manage rules throughout the domain, and can access and modify information about the entire domain.

Rules are intended to be triggered based on a condition in the domain, and to respond to that condition by performing a corresponding action. A rule’s power lies in its ability to react to a situation, allowing domain-wide control of resources and nodes. Therefore, rules are not intended for activities such as polling, which is a function better suited to sensors and triggers.

Here is a tip! If you have never written a rule before, start by developing a simple rule so you can observe its behavior and gain a sense of how you might plan the logic required to support more robust availability requirements.

How to plan a rule

A rule runs to solve a problem. Therefore, before you write a rule, you must identify the problem and then craft an efficient solution that the rule can execute. This requires some preliminary analysis and design in which you come to understand the full scope problem and as well as the solution you are about to create and unleash. Carefully plan how the rule should tackle each element of the problem. In doing so, keep the following general tasks in mind:

1. Understand the behavior of rules in AutoStart.

2. Identify the node(s) or process(es) you want the rule to monitor.

3. Identify the condition(s) that you want the rule to monitor. The condition can be hardware-related, software-related, or some combination of the two.

4. Figure out how you're going to detect each condition. You may need to use a sensor, or detect an event that has occurred if you're working in a Windows environment.



5. For each condition you're going to detect, determine the action that the rule will take. If the rule needs to communicate with and control the application process, you'll need an actuator. If the rule needs to take other actions, you may need a utility process.

6. Design the rule by determining:

a. How many triggers you will need to drive the rule. Design the triggers and the rule so that the triggers work correctly with each other. For example, make sure the rule works correctly if one trigger fires but another does not.

b. What local variables and persistent variables does the rule need?

c. What, if any, process, node, and domain attributes will need to be defined?

7. You may have to consider some of the following questions:

• Do you need more than one rule?

• Does one rule need to run before another?

• Do you need additional sensors to monitor your application’s availability?

These options can be readily resolved by the rule subsystem. There is no limit on how many rules you create nor is there any limit on how interdependent the rules are. As you will see as you create a rule, even resource groups and rules can interact.

Configuring triggers

Once you have determined the condition that you will use to initiate the execution of a rule, you must configure the trigger. To trigger the rule based on a value or state, use a sensor-based trigger; to trigger the rule based on a scheduled time or on demand, use a scheduled trigger. You can also configure a rule to be triggered by any combination of the two trigger types.

The sensor-based trigger is the intermediary between the sensor (which simply reports values) and the rule (which performs actions). The trigger is the mechanism which compares the sensor value to the desired value. If the conditions match, initiates the execution of the rule.

You define the sensor-based trigger to monitor a specific sensor value for the node or process, and the condition that the trigger should check. There are a number of conditions that can be selected, including threshold, absolute change, percent of change, percent change within a specified range and state change. Before you can define a sensor-based trigger, the sensor must be accessible through to AutoStart. The sensor must exist and the process that provides it, if any, must be running.

Note: The rule text is not intended to be used for polling a sensor value. That is the job of a trigger.

A scheduled trigger is defined to fire at a scheduled time. The scheduled time may occur once, or on a defined cycle such as each hour, each day, or each year. The schedule may also be at a specific time on certain days of the week, as your needs dictate. You can also configure the trigger to fire only on demand, based on user input or from within a rule.

Planning a rule 151


Associating triggers with rules

A single trigger can be associated with multiple rules. When that single trigger fires, it initiates each associated rule. A single trigger can also be associated with a number of instances of an application. By correctly configuring the processes’ sensor classes, the trigger can be fired when the condition of the sensors of any monitored processes sensors matches the desired condition.

A single rule can be triggered by one or more triggers. The rule text executes when any of its associated triggers fires. If you want a rule to run only when all of its associated triggers fires, you must specifically code it to do so.

Once you have configured the triggers, you can then associate them with rules. A rule cannot be enabled if there are no triggers associated with it. When associating the triggers with the rule, make sure that you select all applicable triggers. During rule writing, you may also need to reference the name of the trigger in the rule text.

Identifying details

Based upon preliminary analysis and design, perform the tasks illustrated in Figure 3 on page 153:

1. Identify the sensors that provide the necessary input. AutoStart provides a number of sensors through proxy processes and the agent itself. If no AutoStart-supplied sensor is appropriate, create an AutoStart-aware application program with the needed sensors and actuators or modify an existing program to include the needed sensors and actuators.

2. Define the necessary AutoStart resources, such as processes, services, managed IP addresses, data sources, and so on.



Figure 3 Planning a rule

3. Start the AutoStart-aware application program to publish sensors and actuators. An aware application program is always started and managed by AutoStart.

4. Define one or more triggers to monitor the sensors and report when the problem condition occurs.

5. Associate the triggers with the rule.

6. Write the rule text that defines the corrective action to take when the trigger(s) fire.

7. Define the rule in AutoStart.

8. Enable the rule to observe its behavior.

9. Test and debug the rule.

Debugging

It is important once you have written the rule that you test and debug it before running it unattended. AutoStart includes a number of utilities to assist in the debugging process, including syntax checking for minor errors and tracing functionality for more complex debugging.

Planning a rule 153


Where the rule runs

When the rule is enabled, AutoStart randomly selects a rule interpreter from the available primary agents. When the rule is triggered, the rule text is interpreted and executed on the node running the rule interpreter assigned to the rule. Once assigned to the rule interpreter, the rule runs on the same rule interpreter each time it is triggered until the rule is disabled or the rule interpreter fails.

As a result, all of your rules must be node independent, written to take into account the random rule interpreter selection. There is no way to determine when writing the rule the node on which it will run, and no way to know once the rule is started, if it will continue to run on that rule interpreter, as it may be relocated at any time.

The rule text, therefore, should not assume that the rule is running in a particular location, and should not be specialized for running on specific nodes, if possible. All commands should be able to be run on any rule interpreter without any special modifications. If the rule is to access applications or data from a specific location, you must provide the full, absolute path to ensure that the information is accessed correctly.

When the trigger is delivered

When the condition that the trigger monitors matches the desired condition, the trigger fires, and the trigger information is delivered to the rule. This information not only includes the notification that the rule should run, but also information needed to configure the environment under which it runs. AutoStart’s global variables, such as the name of the trigger which fired the rule, the node on which the trigger fired, and more, are assigned values when the trigger is delivered. Once the rule runs, this global information is available to the rule, and can be used within the rule text to direct the actions of the rule. For instance, you could code the rule to perform different actions based on the node that originates the trigger.

Do not confuse performing actions based on a triggering condition from a specific node with coding the rule to run on a specific node. The rule’s actions may be node-specific; its operation must be node independent.

When writing the rule, you should code it so that it is stateless. That is, when a trigger is received by the rule, this trigger should not cause the rule to set a condition which in turn expects another condition. For instance you should not code the rule so that if a particular trigger is received, the rule enters a state where is expects only to receive another specific trigger. The normal operation of the domain may prevent this progression from occurring, and events may be missed if the rule is coded in this manner.

If the rule is to perform an action based on the trigger that fired it, you can use AutoStart-supplied global variables to determine which trigger fired, the node on which the trigger fired, and other useful information.

Rule execution guarantees

Once enabled, a rule gets a chance to run when any of its associated triggers fires, provided at least one primary agent is active. If a rule is driven by more than one trigger, the rule runs when any of the triggers fire. Do not assume that all of the rule’s triggers have fired when the rule executes. The first trigger that fires causes the rule to run. If the rule depends on data from all triggers, you must write it so that it does not execute until the state of all triggers report the required values.



A rule can run on any node that hosts a primary agent. Do not make any assumptions about where the rule runs. A rule must be written to be node independent. If a domain uses both UNIX-based and Windows NT-based nodes for primary agents, each rule must be written so that it runs appropriately on both operating systems.

A rule is guaranteed to run as long as a primary agent is active. However, rules are not guaranteed to run to completion without failure. Rules can be interrupted during their operation. For example, if the node on which the rule is executing fails during rule execution, the rule cannot complete. AutoStart relocates the rule to a rule interpreter on another primary agent, if one is available. If the state that causes the trigger to fire is still true, the rule executes again, executing from the beginning of the rule text.

Resource and node state updates

The AutoStart agents monitor and report on the states of the nodes and managed resources in the AutoStart domain. AutoStart updates the states of any affected node or resource in its database and reports the events to any rules impacted by the change.

When AutoStart detects a node failure it guarantees that a number of events occur before the rule receives a trigger indicating the Failed state:

◆ All previously running processes on the failed node are marked as Failed.

◆ All previously active managed IP addresses and node aliases on the failed node are marked as unassigned.

Later, when the failed node comes back online, AutoStart guarantees that the following event occur before a rule receives a trigger indicating the Running state:

◆ AutoStart identifies the state of all managed processes on that node and updates the state in the AutoStart database.

◆ AutoStart identifies the state of all managed IP addresses and node aliases for the node and updates the state in the AutoStart database.

◆ AutoStart queries the state of all attachable data sources and updates them to reflect the current state, either Attached or Unattached. If the query-state operation on the data source fails, the data source’s state is marked as Unknown. A data source is attachable if the node is included in the valid node list when the data source is defined in the data source console.

If a data source was in the Attached state prior to a node failure, AutoStart recognizes the data source’s state as Attached at node restart even if the attach count exceeds the allowable maximum specified when the data source was configured.

Guidelines for writing rule textA rule runs for a reason that you have identified; it is intended to perform useful functions in an AutoStart domain. Rules are powerful, and flexible, and it is for that reason that you need to follow a number of strict practices when writing a rule. Adhering to these doctrines and practices described in this topic will help ensure that your rules run as intended, in an efficient manner.

Guidelines for writing rule text 155


Note: You can create a rule and manage a rule from the AutoStart console or, optionally, from the CLI interface.

Understand the problem and the conditions that cause it

Having a solid grasp of the problem to be solved is key to planning a rule and its components. Analyze the problem to be solved, and design a solution that addresses the problem.

When writing a rule, it is also important that you understand the conditions that occur in the AutoStart domain that will fire a trigger and cause your rule to run. If you do not understand the condition that will fire the trigger and run the rule, you cannot properly write the rule to resolve the problem, and consequently, the rule may not do what you expect it to do.

Sample

For example, one reason for needing a rule is that you want to relocate a resource groups based on the node that is most highly available, not based on a predetermined preferred nodes list.The subset of nodes on which a resource group can be brought online. So you:

1. Create a nodeless resource group A resource group intended to be controlled from a rule. Because the rule will determine the nodes on which the resource group will run, there are no nodes assigned in the preferred node list. The nodeless resource group is configured, and once assigned to a node by the controlling rule, runs like a standard resource group. See also Resource Group. Nodeless resource groups do not have a specified relocation destination, an can only be started by a rule.

2. Create a rule that determines node to which the nodeless resource group is relocated when a failure occurs.

Review the AutoStart demo and rule tutorial

There are two AutoStart demonstrations you should review before striking out on your own:

◆ Try the rule tutorial before authoring your first rule script. This tutorial is described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

◆ Another demonstration illustrates the basic operation of AutoStart’s rules and some of the capabilities of rules within the domain. It introduces the concepts for rule creation and operation. It first defines example triggers, rules, sensors, and actuators, then enables the rules to manage example AutoStart-aware applications. Source code for the sample aware applications is included. For a description of the Windows and UNIX demonstration rules that are provided with AutoStart, refer to AutoStart online help and the EMC AutoStart Administrator’s Guide.



Get to know Perl

You must write the rule text as a Perl script. Be familiar with the concepts and conventions of Perl programming. You do not need to be a Perl expert to make useful, even sophisticated, rules, but you do need a basic understanding of Perl. You'll need to get yourself a Perl guide, and also refer to AutoStart online help and the EMC AutoStart Administrator’s Guide for basic requirements and helpful tips.

Gain a basic understanding of the AutoStart API extensions you'll be using

Included in AutoStart are different types of AutoStart extensions you can use to give a rule control in the AutoStart domain. How you use them, the information they provide, and the access they provide to the AutoStart database are described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

AutoStart provides over 60 built-in rule API calls, which provide access to the AutoStart database and runtime environment, and allow the node independent interaction between rules and the AutoStart domain. AutoStart subroutines are extensions to standard Perl. Rules that use AutoStart’s built-in library of rule API calls provide:

◆ Centralized administration with node independence

◆ A consistent AutoStart domain-wide view

◆ Direct access to the AutoStart database

Node independence

When properly written, a rule provides node-independent responses to conditions reported anywhere in the AutoStart domain. For example, you don’t need a different rule for each node to monitor and respond to high CPU loads on each machine. The API calls provide AutoStart domain-wide visibility of nodes and resources, management of resource groups, and other advanced functionality which allows you to leverage all of AutoStart’s power directly from your rules.

The result is a management tool which can be configured from a central location that watches the entire AutoStart domain and responds accordingly. This automation facility is a powerful infrastructure that can be applied to a wide range of availability and system automation tasks.

Access to AutoStart's database

Another way rules are different than simple scripts are the API extensions that AutoStart provides. API subroutines allow a rule to perform centralized administration. The API subroutines provide a rule direct access to all of the information stored in AutoStart’s database. Because information is accessed from the AutoStart database, it is consistent, providing accurate, up-to-date information on a domain-wide basis, regardless of the node on which the rule runs.



Remember: rules are single-threaded

Only one rule is allowed to run on a single rule interpreter at one time. If two rules running on the same rule interpreter are triggered to run at the same time, one rule runs, and only when it has completed, then the second rule runs. However, it is possible for rules to be on different rule interpreters, so you should not make any assumptions about where the rule runs.

Although API subroutines and the Perl sleep() function allow AutoStart communication between agent processes, these calls to not allow another rule to run on the rule interpreter. For more information, see “The rule interpreter” on page 138.

Allow the AutoStart agents to run

Do not let the rule block agent-process communication for extended periods of time. Because the AutoStart agent is composed of a set of self-monitoring processes, a rule must not appear to be unresponsive. At certain times during normal operation, many non-AutoStart functions block communication between the agent processes. This can be problematic if a rule performs a non-AutoStart action that takes more than 15 seconds. For example, calling a blocking call that will take some time may cause the agent to declare the rule unresponsive and terminate it. All AutoStart API subroutine calls and the Perl sleep() function are safe to call for extended periods of time as they allow communication with the other agent processes to continue while they execute.

Understand the use for actuators

When including an actuator in a rule, you may want to fire it only when a sensor reads a specific value. In doing so, however, keep in mind that rule text is not intended to be used for polling a sensor value; that is the job of a trigger.

Understand the triggers you are using

Because they are driven by triggers, rules can monitor sensor points from any or all nodes and take action as necessary, or to run at a set time. Any trigger that drives the rule gives the rule access to the conditions that the trigger monitors; as a result, the rule “see” the environment around it, but it sees only those elements of the environment that the trigger gives it access to.

A rule is typically driven by at least one trigger. In fact, a rule can be driven by any number of triggers. Furthermore, a trigger can drive any number of rules. The trigger’s job is to read a sensor or event or clock and wait for a specified condition to occurred in the AutoStart domain. AutoStart supplies many sensors, each which monitors an environmental factor such as a resource state, a Windows event log, or the system clock. If you can't find an AutoStart-supplied sensor that provides the environmental factor you're looking for, you can create your own sensor. Note that you must create any trigger that you use, but you can re-use any trigger from one rule to the next, provided of course that the trigger has a common use across multiple rules. To learn more, see “Triggers” on page 139 and “Sensors” on page 147.



Some rule basics for using triggers

There are some basic practices you need to understand about the triggers you'll be using with rules:

◆ It is critical that you have a thorough understanding of the types of triggers you’ll be using to drive the rule. For example, the rule will also run when any sensor-based trigger that drives it is activated. If you do not take this into account, the rule may cause an unexpected action. For more information, see “Triggers” on page 139.

◆ The rule text is not intended to be used for polling a sensor value. That is the job of a trigger.

◆ Any rule that you want to enable and run must be driven by at least one trigger. A rule can be driven by more than one trigger, and one trigger can drive multiple rules.

◆ A trigger fires when the specified condition matches the trigger’s current condition; when the trigger fires, it runs the rule. If you use conditional statements in your rule, only the portion of the rule matching the current condition is executed.

◆ Before a rule can be enabled, you must define and associate at least one trigger with the rule. In general, a rule must be driven by at least one trigger; a rule that is not driven by a trigger cannot be enabled and never runs. There are only two types of rules that do not need to be associated with a trigger:

• Rules that are in progress.

• Rules used only to store rule text for use with API subroutines, also called triggerless rules. You store and retrieve rule text contained in a triggerless rule by using the ft::SetRuleText and ft::GetRuleText subroutines that you issue from other operating rules.

◆ To pass information from the trigger to a rule, use Perl variables that are initialized and managed by AutoStart. These variables hand over control to the rule, allowing the rule to determine the exact condition reported and its source.

◆ A rule can be written to handle a large class of events from one or more sources. Make sure you write information about the source of the event in a way that allows the rule to be node-independent, so that AutoStart can perform the reaction in the same way regardless of the process or node that generated the event.

◆ In addition to the sensor value and source, the values placed in the variables provide information about the current trigger. Typical information includes the name of the trigger that caused the rule to run, the number of times the rule has run on the current rule interpreter, and the state of the trigger-either On or Off. The rule can access all of this information, and then take action based on the values of this information.

◆ If a rule is driven by multiple triggers, it is your job as the rule programmer to determine which trigger caused the rule to run. In the rule text, you can use global Perl variables provided by AutoStart, such as $ft::CurrentTrigger, %ft::TriggerOn, and %ft::TriggerOff to determine the current state of a trigger.



Generate visible output from a rule

Because the rule interpreter is launched as part of an AutoStart primary agent, it typically does not have a standard output window. Print statements that do not have a designated file handle or print statements sent to the STDOUT file handle are output to the window from which you ran the Perl script. Since the AutoStart rule interpreter typically runs as a background process, any messages sent to the STDOUT file handle are not seen.

For that reason, to generate visible output statements from rules you'll have to use either of the following API subroutines:

◆ ft::PostEvent() - See ft::PostEvent subroutine. Use this subroutine for important messages that go to the AutoStart event log. The ft::PostEvent subroutine posts event messages to the AutoStart event log. Posting events allows you to provide informational, warning and error messages so that the operation and status of the rule can be tracked during operation.

◆ ft::Trace() - See ft::Trace subroutine. Trace messages are used for debugging and tracing rule execution.

Sample

For example the following message sent to output in a script:

print "The resource group was moved successfully.\n";

might become:

&ft::PostEvent($FT::SEV_INFO, "The resource group was moved successfully.");

Note the ampersand (&) at the beginning of the rule API subroutine. All API subroutines are implemented as Perl subroutines, and follow the same rules as Perl subroutines when calling the subroutine, passing arguments, and other operations. For more information, consult a Perl guide of your own choosing.

The first argument to the subroutine, $FT::SEV_INFO is defined by AutoStart to represent the severity level of the event. Similar to #define in C, AutoStart defines many constants in the FT package.

AutoStart's API subroutinesAutoStart subroutines are extensions to standard Perl. When used in rules, API subroutines give you access to information in the AutoStart domain and the AutoStart database directly, and as a result let you take action in the AutoStart domain.



The following API subroutines are available in AutoStart:

Table 1 API Subroutines and descriptions (page 1 of 4)

API subroutine Description

ft::AssignManagedIP Assigns a managed IP address to a node in the AutoStart domain.

ft::AssignNodeAlias Assigns the specified node alias to the specified node.

ft::AttachDataSource Attaches the specified data source to the specified node.

ft::BringGroupOnline Runs the startup sequence for the specified resource group after which the resource group comes online.

ft::DeleteRuleVariable Deletes the specified rule.

ft::DetachDataSource Detaches the specified data source from the specified node.

ft::DisableGroupMonitoring Disables monitoring of the resources in the resource group.

ft::DisableRule Disables the evaluation of a specified rule.

ft::EnableGroupMonitoring Enables monitoring of the resources in a resource group.

ft::EnableRule Enables the evaluation of the specified rule.

ft::ErrorToString Returns a text string associated with an error code.

ft::EvalRule Gets a rule from the AutoStart database and runs it.

ft::FireActuator Fires the specified actuator given in actuator name.

ft::FireOnDemandTrigger Fires the trigger which will pass value, data and a trigger state which is specified to the rules that monitor the trigger.

ft::GetAttributes Retrieves the attributes for an object from the database.

ft::GetAvailTrackingEvents Gets an object’s availability tracking events for the specified period.

ft::GetAvailTrackingInfo Returns the availability tracking information for a resource group.

ft::GetDataSourceInfo Displays definition information about a data source.

ft::GetGroupNodeList Gets a resource group's preferred nodes list.

ft::GetNodeProperties Gets a node’s properties.

ft::GetOSType Returns a bitmap that shows what version of the OS is currently running.

ft::GetRuleLock Gives rules a test and set lock function in which only one rule can successfully get the lock. It will also free this lock if the rule that holds it fails.

ft::GetRuleText Returns the text of the specified rule.

AutoStart's API subroutines 161


ft::GetRuleVariables Gets a rule’s persistent variables.

ft::GetSensorValue Gets the sensor value and sensor data of a sensor.

ft::GetUserData Retrieves the user data from the database.

ft::ListDataSources Lists all defined data sources for the AutoStart domain or assigned to a node, if specified.

ft::ListManagedIPs Lists all of the managed IP addresses in the AutoStart domain.

ft::ListNicsByNode Lists the network interface cards (NICs) that are resident in the specified node in the AutoStart domain.

ft::ListNodeAliases Lists all defined node aliases in the domain.

ft::ListNodes Returns a list of all nodes currently defined for the AutoStart domain and their states.

ft::ListProcs Returns a list of all processes that AutoStart manages. This list includes processes, services, process proxies, and node proxies.

ft::ListProcsByClass Returns an array of all processes for a specified class and two associative arrays that contain the nodes on which the processes last ran or are currently running, and the process states.

ft::ListProcsByNode Returns an array of identifiers for all processes currently or previously running on a specified node along with an associative array containing the state of the processes.

ft::ListResourceGroups Lists all resource groups in the AutoStart domain.

ft::ListRules Returns a list of all rules defined, followed by the state, and then, if running, the node on which the rule is being processed.

ft::OnDisable Registers a clean-up subroutine to be run when a rule is disabled.

ft::Ping Pings a node or remote IP address.

ft::PostEvent Lets a rule interface with the AutoStart event log or error log so you can report errors and informational messages.

ft::QueryDataSourceState Returns the state for the specified data source on the specified node.

ft::QueryRegistryValueDWORD Returns the decimal value of a given 32-bit DWORD Windows NT registry value on a given node.

ft::QueryRegistryValueString Agent's entry point on the node on which its registry will be queried.

ft::ReleaseRuleLock Releases the rule lock if you are the owner.





ft::RelocateGroup Relocates a resource group.

ft::ResetActiveComputerName Reads the ComputerName from the Windows registry and writes the name in the ActiveComputerName value.

ft::SendMail Sends mail by establishing a connection with the SMTP.

ft::SendSNMPTrap Notifies the primary agent group of an SNMP event to be sent.

ft::SetActiveComputerName Takes a passed in name and stores it in the Windows registry ActiveComputerName value.

t::SetAttribute Sets the attributes for an object from the database.

ft::SetTargetNic Changes the target NIC that a managed IP address is associated with on the provided node.

ft::SetRegistryValueDWORD Sets a dword registry entry.

ft::SetRegistryValueString Sets a string registry entry.

ft::SetRuleText Replaces the text of a specified rule.

ft::SetRuleVariable Sets a persistent rule variable for the specified rule from the AutoStart domain.

ft::StartProc Starts the specified process, service, process proxy, or node proxy on the specified node.

ft::StartTelnetSession Replaces the text of a specified rule.

ft::StartUtilProc Starts the specified utility process or Telnet session on the specified node.

ft::StopProc Stops the specified process.

ft::TakeGroupOffline Makes an attempt to take a group offline.

ft::TimeString Returns a text string containing the current time, in the format MM/DD/YY hh:mm:ss (that is, month/day/year hour:minute:second).

ft::Trace Associates a level with one or more trace messages that have been provided as arguments in the rule's.

ft::TraceAssert Causes the text “ASSERTION FAILED $label” to be output as a trace message for debugging the rule if $expression evaluates to false.

ft::TraceEntry Traces given parameters of a rule subroutine and specifies thetrace message to be output for debugging the rule.



AutoStart's API subroutines 163


Use APIs to make processes AutoStart-awareThe EMC AutoStart application programming interface (API) includes a library of functions that let an application process to interact directly with the AutoStart agent. Any process that uses these functions to do so is called an AutoStart-aware process. AutoStart-aware processes are made aware by customizations that you create. This lets you extend the AutoStart environment and your management capabilities.

A process that does not use the AutoStart runtime API is said to be unaware of AutoStart, and so is called an unaware process. This includes third-party and shrink-wrapped software that is monitored by AutoStart. Unaware processes can be associated with a proxy process to allow AutoStart to monitor various conditions.

Note: If the AutoStart-aware process is a Windows service, you must use the ft_ServiceName API function to identify the service to the AutoStart runtime library bound into the service.

An AutoStart-aware process interacts directly with AutoStart to provide sensors and actuators. An AutoStart-aware process can include one or more sensors and actuators and has bound the AutoStart programming library to publish sensors or actuators.

The EMC AutoStart API includes several functions you can use to make a process aware.

Compiling and linking an aware application program

To use the AutoStart functions:

◆ You must compile the application program with the ftsdk.h header file. These files are located in $FT_DIR\sdk\include\.

◆ You must link the program with the AutoStart runtime library ftsdk.lib. These files are located in $FT_DIR\sdk\lib\.

Use a proxy process to publish sensors and actuators

To create an AutoStart-aware process, you create a proxy process, which publishes sensors and actuators on behalf of the process you are making AutoStart aware of. You link the proxy process's sensors and actuators to the process when you define the process to AutoStart. Proxy processes are created in the same way as a standard AutoStart-aware application. For more information, see “Process proxies” on page 129.

ft::TraceExit Traces the output and error code return of a rule subroutine. In the trace output, the values are preceded by the given trace message string.

ft::UnassignManagedIP Removes the managed IP address from the node to which it is currently assigned.

ft::UnassignNodeAlias Removes the specified node alias from node to which it is currently assigned.





When the aware process executes

Figure 4 on page 165 shows the components in the interaction between an executing aware process, an agent, and the AutoStart console.

When the aware process executes, the following sequence of actions takes place:

1. If the process is a Windows-based service, the process calls ft_ServiceName to identify (to AutoStart) the service’s name.

2. The process defines sensor1 and actuator1 by calling the ft_DefineSensor and ft_DefineActuator functions.

3. Optionally, the process calls ft_SetResponseTime to set the maximum number of seconds allowed between calls to ft_Execute.

Figure 4 The aware process executes

4. The process calls ft_Execute for the first time. The function builds the process information (the list of sensors and actuators) and sends the process information to the agent.

Use APIs to make processes AutoStart-aware 165


5. The agent stores the process information in the database. This establishes the aware process's sensors and actuators.

At this point, a user can use the AutoStart console to create a trigger that is based on sensor1 and a rule that uses actuator1. Then the user associates the rule with the trigger (or more than one trigger) and enables the rule. The purpose of this rule is to fire actuator1 when sensor1 reaches the trigger’s defined value.

6. The agent sends the trigger definition to the application.

7. When the process calls ft_Execute again, that function detects the new trigger definition and begins calling the sensor callback function at a specified frequency. When the trigger condition is met, ft_Execute tells the agent and the agent tells the rule.

To give the rule a chance to set up, ft_Execute sends the rule an initial value at the time the rule is enabled.

8. Now the rule fires the actuator.

9. The agent verifies that the actuator exists and determines the process where the actuator should run.

10. If the target process is running, the local agent sends a request to the process.

11. When the process calls ft_Execute next, the function detects the request, looks up the actuator callback function and executes the routine.

Aware application class

An aware application class is a logical naming method that allows you to group processes that have the same functions. When defining a process you must specify an aware application class under which any sensors for the process are published. To aid in writing rules that can manage multiple instances of a process, all like processes should publish their sensors under the same aware application class. For example, three web server processes should each have a common aware application class, such as Web_Server.

AutoStart's global variablesGlobal variables are accessed by rules to get information about the state of the domain at the time the rule was run. A global variable's values, which are populated by AutoStart during normal operation of the domain, provide information relating to triggers, sensors, originating processes and nodes, the number of times the rule has run on the current node, string translation, error codes, and more. These variables provide the mechanism for passing runtime information to a rule. Before evaluating a rule, AutoStart establishes context for the rule by retrieving the values of global variables to get information about the current runtime environment.

Each AutoStart global variable is scalar (containing a single value) or an array (which can contain multiple values). There are two types of arrays available in Perl: standard arrays-referred to simply as arrays-or associative arrays. When writing your rule, be sure you understand what type of variable you are using so that you can use the right syntax. Using improper syntax for the global variable can cause a rule to run improperly.



Referencing a global variable

AutoStart's global variables are part of the internal ft:: package. You must reference a global variable using the appropriate variable identifier, either $ (scalar), @ (array), or % (associative array), followed by ft::. For example:

$ft::TriggerOn{Trigger_Name};

Perl scripts versus rules

An important difference between a standard Perl script and a rule is that a rule is run within the controlled AutoStart environment. Standard Perl script’s variables are saved in memory; but AutoStart's global variables store values to be used later in the AutoStart database. Global variables are defined by the AutoStart rule interpreter and when the rule runs, each global variable's value is established. Once a global variable’s value is established, that value can be used by other statements in the script or rule. The value of the variable can change as the script or rule runs.

Predefined global variables in AutoStart

Before writing a rule, you should familiarize yourself with all the global variables that are available in AutoStart. Below is a list of all of the global variables supplied by AutoStart, along with each variable's use; click on the variable name to read more about the it.

Note: In Perl, variable names are case sensitive. So $MyVar, $myVar, and $myvar are all considered to be different variables. Likewise @Array, @array, %AssocArray, and %assocArray are all different variables.

Global variable Description

Rule-related

$ft::CurrentNode Contains the name of the node on which the rule is currently running.

Translate strings

@ft::DataSourceNodeStateString

Converts the returned constant for the data source state to a string value.a

@ft::ErrorString Converts the returned constant upon execution of a subroutine to a string value.a

@ft::GroupStateString Converts the returned constant for the resource group state to a string value.a

@ft::IPStateString Converts the returned constant for the managed IP address state to a string value.a

@ft::NicStateString Converts the returned constant for the NIC state to a string value.a

@ft::NicUsageString Converts the returned constant for the NIC usage to a string value.a

@ft::NodeAliasStateString

Converts the returned constant for the node alias state to a string value.a

@ft::NodeStateString Converts the returned constant for the node state to a string value.a

AutoStart's global variables 167


Example for combining global variables in a rule

This example demonstrates how you can combine global variables in a rule. This example uses two of AutoStart's global variables:

◆ %ft::TriggerOn

◆ $ft::CurrentTrigger

The example shows you how you can use these two variables in a rule to run the rule when the trigger that fires the rule has a state of ON.

Sample portion of rule GlobalVarTestAssume rule GlobalVarTest is triggered by an on-demand trigger called Demand. This example represents only a portion of a rule:

1 if ($ft::TriggerOn{$ft::CurrentTrigger}) { 2 &ft::PostEvent($FT::SEV_INFO, "$ft::CurrentRule:Rule triggered by $ft::CurrentTrigger.");

@ft::ProcStateString Converts the returned constant for the process state to a string value.a

@ft::RuleStateString Converts the returned constant for the rule state to a string value.a

Trigger-related global variables:

$ft::CurrentRule Contains the name of the rule currently being executed.

$ft::CurrentTrigger Contains the name of the trigger which fired to execute the rule.

$ft::OriginSensorClass

Contains the name of the Sensor Class from which the trigger was fired.

$ft::RunCount Contains the number of times the rule has run since the rule was enabled and running on the current rule interpreter.

%ft::OriginName Contains the name of the process or node whose state caused the trigger to fire.b

%ft::OriginNode Contains the name of the node that caused the trigger to fire.b

%ft::SensorArgument Contains the argument parameter specified in the trigger definition of the current trigger.b

%ft::SensorData Contains the optional data string returned by the specified trigger.b

%ft::SensorName Contains the name of the sensor monitored by the named trigger.b

Global variable Description

%ft::SensorValue Contains the name of the sensor monitored by the named trigger.b

%ft::TriggerOff Evaluates to TRUE if the trigger state is OFF. Only threshold and state comparison trigger can have a value of OFF.b

%ft::TriggerOn Evaluates to TRUE if the trigger state is ON. Only threshold and state comparison trigger values change from ON to OFF.b

%ft::ValueType Contains the state change condition of the trigger.b

a. The variable is an array. Array values are indexed by integer values.

b. The variable is an associative array indexed by trigger name.



3 &ft::PostEvent($FT::SEV_INFO, "$ft::CurrentRule:This rule has run $ft::RunCount time(s) on this ruleinterpreter.");}4 if ($ft::TriggerOff{$ft::CurrentTrigger}) { 5 &ft::PostEvent($FT::SEV_INFO, "$ft::CurrentRule:The value of $ft::CurrentTrigger is OFF."); 6 &ft::PostEvent($FT::SEV_INFO, "$ft::CurrentRule:This rule has run $ft'RunCount time(s) on this ruleinterpreter.");}

Explanation1. To ensure that this portion of the rule runs only when the state of the trigger is ON, an

if statement is used. The if statement checks the current trigger to see if the state is ON. If so, the block of rule text that follows is executed.

2. Note how the ft::PostEvent subroutine is used to print, the value assigned to $ft::CurrentRule to the event log. It also prints the trigger that triggered the rule. In this example, only one trigger can trigger the rule, but in a rule driven by multiple triggers rule text would help you determine which trigger caused the rule to run.

3. Using the $ft::RunCount variable, you can determine the number of times the rule has run in the timeframe since it was enabled on the current rule interpreter.

4. The if statement determines the condition if the trigger state is OFF.

5. If the trigger state is OFF, that information is output to the event log using the ft::PostEvent subroutine.

Event log outputLet’s assume you use the Fire On Demand trigger dialog box in the AutoStart console to trigger the rule three times: twice with a state of ON and once with a state of OFF. As a result, the event log contains the following output:

... Rule GlobalVarTest is enabled on java.

... Running rule GlobalVarTest for trigger Demand (state ON value 15)

... GlobalVarTest: Rule triggered by Demand.

... GlobalVarTest: This rule has run 1 time(s).

... Running rule GlobalVarTest for trigger Demand (state ON value 15)

... GlobalVarTest: Rule triggered by Demand.


... Running rule GlobalVarTest for trigger Demand (state OFF value 63)

... GlobalVarTest: The value of Demand is OFF.


AutoStart's global variables 169

APPENDIX AThe AutoStart Console and the CLI

Invisible Body Tagt

This appendix describes:

◆ The AutoStart console ........................................................................................... 172◆ CLI commands ...................................................................................................... 174

The AutoStart Console and the CLI 171

The AutoStart Console and the CLI

The AutoStart consoleThe AutoStart console provides a centralized interface that allows you to manage and configure objects within your AutoStart domains. The AutoStart console is divided into two panes: the object tree on the left (called the tree view), and the object-information tabs on the right. Each object-information tab has its own Help button, which you can click to get more information specifically for that tab. See, Figure 5.

Figure 5 The AutoStart Console

Versions

Each version of AutoStart comes with its own console. A console can see AutoStart agents of the same version. For example, a 5.3.x console can see any 5.3.x agent, but it can't see a 5.2.x agent. Use a 5.2.x console to see 5.2.x agents.

For that reason, if you have AutoStart domains that have agents from multiple versions of AutoStart, you'll need a console for each version. You can install and launch multiple AutoStart consoles. For more information, see the EMC AutoStart Installation Guide.

Note: The title bar of the AutoStart console tells you the version of AutoStart you can use it for.



States

The console displays the state of resources defined to AutoStart domains. The console automatically keeps the states of resources up-to-date. You do not need to manually refresh these states. Note that states that AutoStart assigns to each resource are described in AutoStart online help and the EMC AutoStart Administrator’s Guide.

Features of the console

To acquaint yourself with features of the AutoStart console, see the following topics:

◆ Icons and buttons in the AutoStart console

◆ Menu actions

◆ Keyboard navigation

◆ The tree view of the AutoStart console

◆ Working with tables

Using AutoStart help

AutoStart online help contains all you need to know about AutoStart except the installation process (which is documented in the EMC AutoStart Installation Guide) and EMC AutoStart Release Notes. You can access AutoStart online help from the AutoStart console by selecting a menu item from the Help menu, or by clicking a screen's Help button.

AutoStart online help includes descriptions of:

◆ A comprehensive search engine

◆ A table of contents that organizes all online documentation

◆ AutoStart concepts and procedures

◆ Field descriptions for all tabs and dialog boxes in the AutoStart console

◆ Help for setting up all modules and plug-ins, such as licensed data sources, and the wizards that implement them

◆ CLI command syntax

◆ Syntax for all subroutines, functions, global variables, and AutoStart-supplied sensors

◆ Scripting guidelines for rules and triggers

◆ Troubleshooting guidelines

◆ A glossary

Once you have launched AutoStart help, you can browse the table of contents from the Contents tab, or search help for any number of terms or phrases by using the Index and Search tabs. You can even print help pages.

The AutoStart console 173


AutoStart plug-ins

For some of the more commonly-monitored applications, AutoStart provides plug-ins you can use to set up those applications quickly and easily in AutoStart. Modules provide wizards with which you can quickly and easily set up of AutoStart resource groups for commonly-used applications. These integrated plug-ins let you enter pertinent data about installed applications.

Plug-ins that require a license

AutoStart provides plug-ins for the following applications, which are available from EMC for additional license keys:

◆ Microsoft Exchange

◆ SQL Server

◆ Oracle

◆ SRDF data sources

◆ MirrorView data sources

Free plug-ins

However, there are two wizards that are readily available in AutoStart to help you set up resource groups for:

◆ Microsoft Windows Print Services

◆ Sites running on Microsoft IIS

CLI commandsYou can use the commands listed below in the AutoStart command line interface (CLI). Commands are not case-sensitive; you can issue them using any combination of lowercase or uppercase letters. But names of resources may be case-sensitive.

All CLI commands and their aliases (that is, their abbreviated shortcuts) are provided below. Not every CLI command has an alias. Each is described in AutoStart online help and in the EMC AutoStart Administrator’s Guide.

! =help= | restartNodeabortGroupabortInstanceaddDomainAttr | adaaddGroupAttr | agaaddLicenseaddManagedIP | addipaddNodeaddNodeAlias | addnaaddNodeAttr | anataddProcAttr | apaassignManagedIP | assignip | aipassignNodeAlias | anaasyncNotifyattachDataSource | atdsbackup


APPENDIX BQuick References

t

This appendix lists the console menu items, CLI commands, and API subroutines for handling resource in an AutoStart domain.

◆ Quick guide: AutoStart domains............................................................................ 180◆ Quick guide: Nodes and agents............................................................................. 181◆ Quick guide: Data sources..................................................................................... 182◆ Quick guide: NICs and NIC groups ......................................................................... 182◆ Quick guide: IP addresses..................................................................................... 183◆ Quick guide: Node aliases..................................................................................... 183◆ Quick guide: Modules and module instances ........................................................ 184◆ Quick guide: State monitors.................................................................................. 185◆ Quick guide: Processes ......................................................................................... 186◆ Quick guide: Services............................................................................................ 187◆ Quick guide: Utility processes ............................................................................... 188◆ Quick guide: Process proxies................................................................................. 188◆ Quick guide: Node proxies .................................................................................... 189◆ Quick guide: Rules, tracing, and debugging........................................................... 190

Quick References 179

Quick References

Quick guide: AutoStart domainsTable 2 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with AutoStart domains.

Table 2 AutoStart domain commands

To do this...You can use this AutoStart console tab, menu item, or tree: You can use this AutoStart CLI command:

From a rule, you can use this subroutine:

Open or Remove Action > Open new domainAction > Remove current domain

ftcli -domain

Connect or Disconnect Action > Connect to domainAction > Disconnect from domain

ftcli -domain

Import Action > Import domain information importDef | import

Export Action > Export domain information export

Back up backup

Restore restore

Grant or Modify a user's access

domain Licensing/Security tab createUser

Remove a user's access domain Licensing/Security tab deleteUser

List users domain Licensing/Security tab listUsers | lu

Get failure detection information

domain Statistics/Domain Failure Detection tab

getFailureDetectionInfo

Set failure detection interval


setFailureDetectionInterval

Set failure detection multicast address


setFailureDetectionMulticastAddress

Set failure detection port domain Statistics/Domain Failure Detection tab

setFailureDetectionPort

Set failure detection time domain Statistics/Domain Failure Detection tab

setFailureDetectionTime

Get a log file domain Event Log tab -or-View > Event log

getLogFile

Post an event Right-click Triggers in the tree view > Create New Trigger > Event Log

ft::PostEvent

Watch events domain Event Log tab -or- View > Event log

watchEvents ft::PostEvent

Add an attribute domain Settings tab addDomainAttr | ada ft::SetAttribute

Delete an attribute domain Settings tab

List attributes domain Settings tab listDomainAttr | lda ft::GetAttributes

Delete user data deleteUserData | dud

Set user data ft::SetUserData


Quick References

Quick guide: Nodes and agentsTable 3 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with AutoStart agents and nodes in AutoStart domains.

To do this...You can use this AutoStart console tab, menu item, or tree: You can use this AutoStart CLI command:


List user data listUserData | lud ft::GetUserData

Add a license domain Licensing/Security tab addLicense

Get a license domain Licensing/Security tab getLicense

List licenses domain Licensing/Security tab listLicenses

List products listProducts

Get the domain's status domain Statistics/Domain Failure Detection tab

status

Table 2 AutoStart domain commands (continued)

Table 3 Node and agent commands

To do this...You can use this console tab, menu item, or tree:

You can use this CLI command:


Add Action > Create new node addNode

Demote Action > Demote node demoteNode | demoteAgent

Promote Action > Promote node promoteNode | promoteAgent

Get Select it in the Nodes tree view getNodegetNodeStategetNodeStatsstatus

ft::GetNodeProperties

List Expand (+) the Nodes tree listAgents | lalistNodes | ln

ft::ListNodes

Delete Action > Delete current mode deleteNode

Mirroring Failure Detection and Mirroring tab setMirrorAddresssetMirrorPartner

Shut down Action > Shutdown agent shutdown

Restart the agent =help= | restartNode

Add attribute Node Properties tab addNodeAttr | anat ft::SetAttribute

List/get attributes Node Properties tab ft::GetAttributes

Query registry settings ft::QueryRegistryValueDWORDft::QueryRegistryValueString

Set Windows registry settings

Use the registry monitor ft::SetRegistryValueDWORDft::SetRegistryValueString

Get OS Node Properties tab ft::GetOSType

Quick guide: Nodes and agents 181

Quick References

Quick guide: Data sourcesTable 4 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with data sources in AutoStart.

Quick guide: NICs and NIC groupsTable 5 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with network interfaces (NICs) and NIC groups in AutoStart.

Table 4 Data source commands

To do this...You can use this console tab, menu item, or tree: You can use this CLI command:


Add Action > Create new data sourceAction > Copy a data source

--

Attach Action > Attach data source attachDataSource | atds ft::AttachDataSource

Detach Action > Detach data source detachDataSource | dtds ft::DetachDataSource

Delete Action > Delete current data source deleteDataSource | dds --

Get Select it in the Data Sources tree getDataSource | gdsgetDataSourceInfo | gi

ft::GetDataSourceInfo

List Expand (+) the Data Sources tree listDataSources | ldslistDataSourcesByNode | ldsn

ft::ListDataSources

Issue a command issueDataSourceCmd | idc

Query Action > Query data source queryDataSource | qds ft::QueryDataSourceState

List a data source’s shares listShares

Delete a data source’s shares

deleteShares

Table 5 NIC and NIC group commands

To do this... To a...You can use this console tab, menu item, or tree: You can use this CLI command:


Add NIC group Action > Create new NIC group --

Change usage NIC Action > Change NIC usage to… modNicUsage | mnu

Reset state NIC Action > Reset state --

Get NIC Select it in the NICs tree getNic | gnic

Get NIC group Select it in the NIC Groups tree getNicGroup | getng

List NIC Expand (+) the NICs tree listManagedNics> | lniclistNicsByGroup | lnbglistNicsByNode | lnbn

ft::ListNicsByNode


Quick References

Quick guide: IP addressesTable 6e lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with IP addresses in AutoStart.

Quick guide: Node aliasesTable 7 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with node aliases in AutoStart.

List NIC group Expand (+) the NIC Groups tree listNicGroups | lng --

Delete NIC group Action > Delete NIC group deleteNicGroup | delng --

Set target NIC setTargetNic | stn ft::SetTargetNic

Table 5 NIC and NIC group commands

Table 6 IP address commands



Add Action > Create new IP address addManagedIP | addIP --

Assign Action > Assign IP assignManagedIP | assignip | aip ft::AssignManagedIP

Unassigning Action > Unassign IP unassignManagedIP | unassignIP | uip ft::UnassignManagedIP

Get Select it in the IP Addresses tree getIP

List Expand (+) the IP Addresses tree listManagedIPs | lip ft::ListManagedIPs

Delete Action > Delete current IP address deleteManagedIP | deleteip --

Ping an address ping ft::Ping

Table 7 Node alias commands



Add Action > Create new node alias addNodeAlias | addna

Assign Action > Assign node alias assignNodeAlias | ana ft::AssignNodeAlias

Unassign Action > Unassign node alias unassignNodeAlias | rna ft::UnassignNodeAlias

Delete Action > Delete node alias deleteNodeAlias | dna

List Expand (+) the Node Aliases tree

listNodeAliases | lna ft::ListNodeAliases

Quick guide: IP addresses 183

Quick References

Quick guide: Modules and module instancesTable 8 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with AutoStart modules.

Table 8 Modules and module instance commands


Add a module instance From the Modules tree, select Action > Create instance for...

Abort a module instance From the Modules tree, select Action > Abort abortInstance

Bring a module instance online

From the Modules tree, select Action > Bring online

startInstance

Take a module instance offline

From the Modules tree, select Action > Take offline

stopInstance

Delete a module instance From the Modules tree, select Action > Delete current instance

deleteInstance

Disable module instance monitoring

From the Modules tree, select Action > Disable failure detection

disableInstanceFailureDetection | difd

Enable module instance monitoring

From the Modules tree, select Action > Enable failure detection

enableInstanceFailureDetection | eifd

Relocate a module instance From the Modules tree, select Action > Relocate instance

relocateInstance

Get module instance information

Select the module instance in the Modules tree getInstance

List module instances Expand (+) the Modules tree, expand the module's tree

listInstanceslistInstancesByNodelistInstancesByParent

Add module addModuleaddModulePatch

Delete module deleteModuledeleteModulePatch

Get module getModulegetModuleJar

List modules Expand (+) the Modules tree listModules

Set module info setModuleLicensesetModuleJarsetModuleRelease

Add node to module addNodeToModule

Remove node from module removeNodeFromModule

Components getComponentdeleteComponentlistComponents


Quick References

Quick guide: Resource groupsTable 9 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with resource groups in AutoStart .

Quick guide: State monitorsTable 10 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with existence monitors and response monitors in AutoStart

Table 9 Resource group commands



Abort Action > Abort abortGroup | ag

Add Action > Create new resource group

Bring online Action > Bring online bringGroupOnline | bgo ft::BringGroupOnline

Take offline Action > Take offline takeGroupOffline | tgo ft::TakeGroupOffline

Delete Action > Delete current resource group deleteResourceGroup | drg

Disable monitoring

Action > Stop monitoring disableGroupMonitoring | dgm ft::DisableGroupMonitoring

Enable monitoring Action > Monitor resource group enableGroupMonitoring | egm ft::EnableGroupMonitoring

Get Select it in the Resource Groups tree getResourceGroup | grg ft::GetGroupNodeList

List Expand (+) the Resource Groups tree listResourceGroups | lrg ft::ListResourceGroups

Relocate Action > Relocate resource group relocateGroup | rg ft::RelocateGroup

Add attribute Advanced tab addGroupAttr | aga ft::SetAttribute

Delete attribute Advanced tab delGroupAttr | dga

List attributes Advanced tab listGroupAttr | lga ft::GetAttributes

Get availability info

Availability Tracking tab getAvailTracking | gat | getAvailInfo | gai

ft::GetAvailTrackingEventsft::GetAvailTrackingInfo

Table 10 State monitors


Add Action > Create new state monitorAction > Copy current state monitor

Delete Action > Delete current state monitor deleteStateMonitor

Get Select the state monitor in the tree view getStateMonitor | gsm

List Expand (+) the State Monitors tree listStateMonitors | lsm

Quick guide: Resource groupsQuick guide: State monitors 185

Quick References

Quick guide: ProcessesTable 11 lists tools available in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with processes in AutoStart.

Table 11 Process commands

To do this... You can use this console tab, menu item, or tree:

You can use this CLI command: From a rule, you can use this subroutine:

Define a process Action > Create new processAction > Copy current process

Define a process proxy Action > Create new configurationAction > Copy configuration

Delete a process Action > Delete current process deleteProc

Delete a process's configuration

Action > Delete current configuration deleteConfig

Get a process Select the process in the Processes tree getProc

Get a process's configuration

Select the process’s configuration in the tree

List processes Expand (+) the Processes tree listProcs | lp ft::ListProcsft::ListProcsByClassft::ListProcsByNode

List a process's configuration

Expand (+) the process's tree

Start a process Action > Start process startProc | start ft::StartProc

Stop a process Action > Stop process stopProc | stop ft::StopProc

Delete sensor deleteSensor

List sensors Process Sensors tab listSensors

Get sensor values and data

Process Sensors tab sensorVal | getSensorVal ft::GetSensorValue

Get sensor history getHistoricalSensorHistorygetHistoricalSensorHistorybyProcgetRecentSensorHistorygetRecentSensorHistoryByProc

List actuators Process Actuators tab listActuators

Delete actuator deleteActuator

Fire actuators Process Actuators tab fireActuator ft::FireActuator

Add attributes Process Settings tab addProcAttr | apa ft::SetAttribute

Get attributes Process Settings tab getProcAttrs | gpa ft::GetAttributes

Delete attributes Process Settings tab delProcAttr | dpa

Defining environment variables

Process configuration Options tab

List process classes listProcClasses


Quick References

Quick guide: ServicesTable 12 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with services in AutoStart.

Table 12 Service commands



Add Action > Create new serviceAction > Copy service

Delete Action > Delete service deleteProc

Get Select the service in the Services tree

getProc

List Expand (+) the Services tree listProcs | lplistServices | ls

ft::ListProcsft::ListProcsByClassft::ListProcsByNode

Start Action > Start service startProc | start ft::StartProc

Stop Action > Stop service stopProc | stop ft::StopProc


List sensors Service Sensors tab listSensors


Service Sensors tab sensorVal | getSensorVal ft::GetSensorValue


List actuators Service Actuators tab listActuators


Fire actuators Service Actuators tab fireActuator ft::FireActuator

Add attribute Service Settings tab addProcAttr | apa ft::SetAttribute

Get attributes Service Settings tab getProcAttrs | gpa ft::GetAttributes

Delete attributes Service Settings tab delProcAttr | dpa


Quick guide: Resource groupsQuick guide: Services 187

Quick References

Quick guide: Utility processesTable 13 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with utility processes in AutoStart.

Quick guide: Process proxiesTable 14 lists tools available in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with process proxies in AutoStart.

Table 13 Utility process commands




Create a standard utility process

Action > Create new utility process

Create a Telnet utility process Action > Create new telnet process

Delete Action > Delete utility process deleteUtilProc | dup

Get Select the utility process in the Utility Processes tree

getUtilProc | gup

List Expand (+) the Utility Processes tree listUtilProcs | lup

Start Action > Start utility process startUtilProc | sup ft::StartUtilProcft::StartTelnetSession

Define environment variables using the console

Utility process Options tab

Table 14 Process proxy commands




Add a process proxy Action > Create new process proxyAction > Copy process proxy

Add a configuration for a process proxy

Action > Create new configurationAction > Copy configuration

Delete a process proxy Action > Delete proxy process deleteProc

Delete a process proxy's configuration

Action > Delete current configuration deleteConfig

Get a process proxy Select the proxy in the Process Proxies tree getProc

Get a process proxy's configuration Select the process proxy’s configuration in the tree

List process proxies Expand (+) the Process Proxies tree listProcs | lp ft::ListProcsft::ListProcsByClassft::ListProcsByNode

List a process proxy's configurations

Expand (+) the process proxy's tree


Quick References

Quick guide: Node proxiesTable 15 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with node proxies in AutoStart.

Start a process proxy Action > Start proxy process startProc | start ft::StartProc

Stop a process proxy Action > Stop process proxy stopProc | stop ft::StopProc

Define environment variables Process proxy configuration Options tab

Table 14 Process proxy commands

Table 15 Node proxy commands



Get Select the node proxy in the Node Proxies tree

getProc

List Expand (+) the Node Proxies tree listProcs | lp ft::ListProcsft::ListProcsByClassft::ListProcsByNode

Start Action > Start node proxy startProc | start ft::StartProc

Stop Action > Stop node proxy stopProc | stop ft::StopProc


List sensors Node proxy Sensors tab listSensors


Node proxy Sensors tab sensorVal | getSensorVal ft::GetSensorValue


List actuators Node proxy Actuators tab listActuators


Fire actuators Node proxy Actuators tab fireActuator ft::FireActuator

Add attribute Node proxy Settings tab addProcAttr | apa ft::SetAttribute

Get attributes Node proxy Settings tab getProcAttrs | gpa ft::GetAttributes

Delete attributes Node proxy Settings tab delProcAttr | dpa

Define environment variables using the console

Node proxy Options tab


Quick guide: Resource groupsQuick guide: Node proxies 189

Quick References

Quick guide: Rules, tracing, and debugging

Rules and triggers

Table 16 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with rules and triggers in AutoStart.

Table 16 Rule and trigger commands



Create a rule Action > Create new rule ft::SetRuleText

Delete a rule Action > Delete current rule deleteRule

Disable a rule Action > Disable rule disableRule | dr ft::DisableRule

Enable a rule Action > Enable rule enableRule | er ft::EnableRule

Evaluate a rule ft::EvalRule

Get a rule Select the rule in the Rules tree getRule | gr ft::GetRuleText

List rules Expand (+) the Rules tree listRules | lr ft::ListRules

Lock a rule ft::GetRuleLockft::ReleaseRuleLock

Get rule variables Rule Settings tab getRuleVars | grv ft::GetRuleVariables

Delete a rule variable Rule Settings tab delRuleVar | drv ft::DeleteRuleVariable

Edit a rule variable Rule Settings tab setRuleVar | srv ft::SetRuleVariable

Create a trigger Action > Create triggerAction > Copy trigger

Post an event using an event log trigger

Right-click Triggers in the tree view > Create New Trigger > Event Log

ft::PostEvent

Get a trigger Select the trigger in the Triggers tree getTrigger

List triggers Expand (+) the Triggers tree listTriggers | lt

Delete a trigger Action > Delete trigger deleteTrigger

Fire an on-demand trigger Action > Fire trigger fireOnDemandTrigger | fodt | signalTrigger | sigtrig

ft::FireOnDemandTrigger

General use ft::ErrorToStringft::OnDisableft::Pingft::TimeStringft::SendMailft::SendSNMPTrap


Quick References

Tracing

Table 17 lists the tools available to you in the AutoStart console, the AutoStart command-line interface (CLI), and in subroutines for working with triggers in AutoStart.

Table 17 Tracing commands

To do this... You can use this console tab, menu item, or tree:You can use this CLI command:


Trace logic ft::Traceft::TraceAssertft::TraceEntryft::TraceExit

Set up/unset a rule’s trace level

rule Settings tabresource group Advanced tab

setTrace | setTraceComponent | traceComponentunsetTrace | unsetTraceComponent

Set up a rule’s trace scope setTraceEventsetTrace | setTraceComponent | traceComponentunsetTrace | unsetTraceComponent

Set up a rule’s trace output

rule Settings tabresource group Advanced tab

setTraceOutput

View trace messages domain’s Event Log tab ruleTrace trace (node)

Other: If a rule’s ft_traceoutput is stdout, it goes to the standard output of the agent process, whatever that might be. Go to that.If a rule’s ft_traceoutput is a file, it writes to a file in the rules directory.

Get dumps dumpCmTabledumpLocalAgentListdumpMemdumpPartitionMapdumpProcTabledumpStateMonInfodumpTraceBuffer | dumpTracedumpTriggers

Quick guide: Resource groupsQuick guide: Rules, tracing, and debugging 191

Quick References


GLOSSARY

This glossary contains terms related to disk storage subsystems. Many of these terms are used in this manual.

A

actuator An external function call in an AutoStart-aware process. Actuators can be invoked from a rule or the AutoStart console. The process that the actuator takes action upon need not be the same process whose sensor value fired the trigger.

agent A self-monitoring set of processes that performs all of AutoStart’s monitoring and management capabilities. The AutoStart agent has two mechanisms for handling event: resource groups and rules. See also “primary agent,” “secondary agent.”

APIs (applicationprogramming interfaces)

See “rule APIs.”

attributes Name/value labels that be attached to the domain, resource group, nodes, or processes. Can be used by rules to organize objects into subgroups, or attach user-defined information to objects for use by rules.

auto-failback An option in a resource group that results in the resource group being taken offline, then brought online on the failback node if it is currently hosted on a node other than the failback node when the failback node comes online.

AutoStart backbone See “backbone.”

AutoStart domain See “domain.”

AutoStart console The graphical user interface that provides users access to create and manage cluster objects managed by AutoStart. Compare to “command-line interface (CLI).”

aware process A process defined to AutoStart that has bound the AutoStart programming library to publish sensors or actuators. Compare to “unaware process.”

backbone The processes running on AutoStart primary agents that provide messaging services. The backbone includes the process monitor, rule interpreter, and the AutoStart replicated database.

B

bring online Initiate an attempt to execute a resource group’s startup sequence. When the startup sequence successfully completes, the resource group is in the Online state.

checkpoint See “data source checkpoint,” “Registry checkpoint.”

C

CLI See “command-line interface (CLI).”


Glossary

command-line interface(CLI)

The command-line interface program provides the same functions as the AutoStart console, but lets you execute commands from within scripts or in batch mode, providing capabilities for debugging and batch processing. Also referred to as the CLI.

Composite data source An AutoStart data source, available on Windows installations, which combines individual data sources into a single data source that can work with three or more nodes. For more detail, see “Composite data sources” on page 69.

console See “AutoStart console.”

D

data source A resource that provides data content to an application. With AutoStart, data sources can be configured to detach from a failed node and attach to a node that takes over the application associated with the data source. The data sources supported by AutoStart are listed in Chapter 4, “Data Sources,”

data source checkpoint A data source checkpoint makes AutoStart wait for the multiple data sources that precede the checkpoint to attach before AutoStart can continue with the startup sequence. If the startup sequence attaches multiple data sources simultaneously (using a parallel attach), the checkpoint waits for all of the data sources in the parallel attach to attach before continuing with the startup sequence.

delay A delay lets you create a buffer of time, to give a process or service or other object time to completely start up or shut down before the sequence continues to the next resource in the resource group's sequence. With a delay you specify exactly how long the startup or shutdown sequence is to wait.

Compare to “data source checkpoint,” “Registry checkpoint.”

disable a rule Turn off a rule and its triggers.

disable monitoring Turn off resource group monitoring. When disabled, AutoStart does not restart resource objects if they go offline. This mode is useful for manual intervention and maintenance of resource objects without the need to relocate the entire resource group.

domain Also known as AutoStart domain, a group of nodes that participate in a particular management scope, configured to work together to provide a highly available environment. An AutoStart domain can have as few as 2 nodes, or as many as 5 primary nodes and an additional 5 secondary nodes. A node can belong to one, and only one, AutoStart domain. All nodes on a network with an AutoStart agent installed are known collectively as the domain. All of the nodes contained within the AutoStart domain share certain characteristics, such as a common replicated database, event log, and failure detection settings. The AutoStart domain should not be confused with the Windows networking domain.

domain communication Inter-agent messaging in AutoStart.

domain network The network AutoStart agents use to communicate with each other. AutoStart clusters can be configured to use an optional independent network or to use the existing service network.


Glossary

E

EMC Mirroring forWindows

A block-level (volume-level) mirroring device for Windows servers, built into AutoStart. Its purpose is to replicate data from a source drive to a target drive; its mirroring is synchronous and is supported only between two nodes. No separate license is required.

enable monitoring Turn on resource group monitoring. When enabled, AutoStart attempts to keep the resource group and all of its resource objects online in the event the object goes offline.

events and rules engine Includes the following AutoStart components: sensors to monitor the resources in the environment, triggers to initiate a response to received sensor data, and actuators to cause the appropriate actions to occur.

existence test A test to check for the physical existence of one or more managed processes. AutoStart provides the option of user-defined existence monitors for processes which can provide a more detailed and accurate status of the process.

G

gatekeeper A small logical volume on a Symmetrix storage subsystem used to pass commands from a host to the Symmetrix storage subsystem. Gatekeeper devices are configured on standard Symmetrix disks.

gigabyte (GB) 109 bytes.

F

fail over See “relocate.”

failure detection Settings for heartbeat interval and multicast or point-to-point addresses used by the AutoStart failure detection mechanism.

fire a trigger Make a trigger's condition true so that it executes. The execution of a trigger is often referred to as firing. A trigger fires when the condition it is defined to recognize occurs. The type of condition a trigger waits for depends on the type of trigger.

ft_ProcState Each managed process or service has one built-in sensor called ft_ProcState. This sensor can be used to monitor the state of the service or process.

G

global variable Global variables provide information relating to triggers, sensors, originating processes and nodes, the number of times the rule has run on the current node, string translation, error codes, and more. These variables provide the mechanism for passing runtime information to a rule. Before writing a rule, familiarize yourself with all the global variables that are available in AutoStart.

H

heartbeat A periodic message sent from agent to agent for managing node state. Also a message sent from an Aware Process to the AutoStart agent to indicate its responsiveness. Heartbeats are used to determine node failure and isolation.

heterogeneousenvironment

EMC provides automated availability products designed for heterogeneous environments (Windows, UNIX, and Linux). AutoStart provides automated application management and


Glossary

availability for large-scale, heterogeneous deployments. AutoStart supports heterogeneous operating systems and data storage options, and both local- and wide-area networks.

I

I/O redirection A system which allows input and output to and from a managed process which would normally go to the file handles STDIN, STDOUT, and STDERR to be redirected to an alternate destination. Since AutoStart usually runs managed processes as background processes, the information to the locations may be lost without redirection.

instance See “module instance.”

isolation detection Isolation detection is a mechanism which allows an AutoStart node to determine if it has become isolated on the network. Its primary purpose is to protect against a situation when a node cannot communicate with any resource. Isolation detection works in conjunction with the Minimum Detection Time set in the Domain Settings.

isolation script The script that is invoked on an AutoStart agent when the node detects that it is isolated. This script includes the instructions that the node follows to ensure that the resource objects, resource groups, and rules on both the isolated node and on other nodes in the domain respond appropriately to the isolation.

M

MAC address Media access control address. A 48-bit address assigned by the interface card manufacturer. Most manufacturers specify a unique MAC address for each card. MAC addresses can be modified by AutoStart, if the model of network interface card allows it, when managed IP addresses are moved among interface cards.

managed A resource is managed if it is defined as an object in AutoStart so that AutoStart has control of the node where it is used. If you want AutoStart to manage a resource, you must define it to AutoStart and configure it so that AutoStart can start and stop it, and relocate it to viable node when necessary.

managed IP address The virtual IP address that AutoStart assigns to a node it is relocating resources to. Each resource group has its own managed IP address that is represented by either: an IP address in n.n.n.n format, or a name service entry that is associated with an IP address in the DNS server.

managed process A process or service that has been defined to AutoStart and will be started and stopped under the control of AutoStart from the AutoStart console, a rule, or a resource group.

managed resources See “resources.”

module An AutoStart plug-in tool lets you make an application highly-available by helping you to quickly set up a resource group that manages a specific application. EMC offers AutoStart modules for the more frequently-used applications that are available. AutoStart modules typically require you to obtain a license key from EMC for their use.

module instance A resource group that has been created using an AutoStart module. You can include module instances in the AutoStart console's Resource Group tree by using the console in


Glossary

advanced-user mode. Note that only CLI commands for instances, not resource groups, work for module instances.

N

network interface card See “NIC (network interface card).”

NIC (network interfacecard)

The hardware component that allows the node to communicate over the network. IP addresses are assigned through software to the network interface card, and those addresses are used to determine the source and destination of network communication. Also referred to as a NIC.

NIC group A mechanism used by AutoStart to group NIC cards using the same subnet. Once included in a group, NIC management tasks such as NIC testing and NIC to NIC failover can be performed.

NIC-to-NIC failover A failover scheme which allows all IP addresses assigned to a network interface card (NIC), including subnet and MAC address information if any, to move to another NIC on the same node if the first NIC fails.

node A system, such as a machine or partition, on which the AutoStart agent is installed.

node alias A name that can be added to a Windows node that NetBIOS can recognize, allowing access from computers on the network. Called a Virtual Server by MSCS, the AutoStart node alias name is not registered with DNS or WINS. Only accessible from NetBIOS clients.

node proxy An aware proxy process provided by AutoStart that publishes node-related sensors and actuators on behalf of a node. See also “process proxy.”

node state The current condition of a node in an AutoStart domain. For detailed information, see AutoStart online help or the EMC AutoStart Administrator’s Guide.

nodeless resource group A resource group intended to be controlled from a rule. Because the rule will determine the nodes on which the resource group will run, there are no nodes assigned in the preferred node list. The nodeless resource group is configured, and once assigned to a node by the controlling rule, runs like a standard resource group. See also “resource group.”

P

parallel attach A startup sequence option that causes the resource group to begin the attach operation and immediately move on to the next step in the startup sequence without waiting for the data source to transition to the Attached state. Its purpose is to let you reduce the time it takes to attach data sources by allowing a resource group's data sources to attach simultaneously.

persistent variable A variable that is maintained in the AutoStart replicated database rather than in local memory on the node. It is available even if the rule referencing it fails.

physical IP address The IP address that is manually assigned to a network interface card (NIC) by a user. This IP address is usually one of the primary addresses used by the NIC. Once in place, this IP address can be managed by AutoStart. See also “virtual IP address,” “NIC (network interface card).”

preferred nodes The subset of nodes on which a resource group can be brought online.


Glossary

primary agent A node running an agent, process monitor, and rule interpreter which provide the main activities for management of a cluster. A primary agent also maintains a copy of the replicated database. At least one primary agent must be running at all times for AutoStart to function properly. There are typically two to five primary agents in a domain. Any domain of two or more nodes should include at least two primary agent nodes. Other nodes in a domain should run as secondary agents. See also “secondary agent,” “agent.”

process monitor Monitors the state of processes and services on a node, and has the secondary function of managing the node failure detection mechanism using heartbeats. Also referred to as ProcMon.

process proxy A proxy process provided by AutoStart that publishes process-related sensors and actuators on behalf of a managed process. The Process Proxy is associated with the process in AutoStart’s process configuration and publishes sensors to the given sensor class. See also “proxy process.”

process states The current condition of a process in an AutoStart domain. For detailed information, see AutoStart online help and the EMC AutoStart Administrator’s Guide.

properties Static characteristics of a node, such as OS version, physical memory, and so on.

proxy process Allows an application to publish actuators and sensors on behalf of another (unaware) process in a tightly integrated manner. See also “node proxy,” “process proxy,” “managed process.”

publish Make sensor data available to AutoStart.

R

Registry checkpoint An AutoStart utility process that is a component of Registry mirroring. The Registry checkpoint blocks the execution of registry operations until Registry mirroring completes import operations during the resource group's startup sequence. The Registry checkpoint is similar to a delay in that it waits for the Registry mirroring process to complete its import operations before allowing the startup sequence to proceed.

Registry mirroring An optional AutoStart tool you can use for mirroring an application's Windows Registry keys to a node before AutoStart relocates the application's resource group to the node. This tool is useful if Registry keys must be, but are not typically, current on a node where the application does not usually run.

relocate Move a resource group (or a module instance) from the node where it is running to another node in the preferred nodes list. During relocation, all managed objects in the resource group relocate to the same node. AutoStart relocates a resource group when the node where it is running becomes unavailable. AutoStart Administrators and Operators can use the AutoStart console to relocate a resource group at any time, for the purpose of doing maintenance or to balance the load on nodes in the AutoStart domain. Synonyms: migrate, fail over, and restart.

replicated AutoStartdatabase

Internal AutoStart database where objects and objects’ states are kept. Each primary agent maintains a synchronous copy of this database.

Reserved NIC An AutoStart usage for a NIC that designates the NIC as not available for NIC failover. When you set a NIC’s usage to Reserved, you are setting the NIC aside for a specific IP address assignment configuration. Therefore it is not used for standard IP address assignment. AutoStart does not automatically include a Reserved NIC in a NIC group. However, AutoStart tests a Reserved NIC for response if the NIC is included in a NIC group.


Glossary

resource group The collection of all the managed objects (such as services, processes, IP addresses, data sources, and others) necessary to keep an application available. The resource group is AutoStart’s primary mechanism for implementing failover and failback. Each resource group coordinates the migration of a set of related resources across an AutoStart environment. Resource groups provide a standard, easy-to-define solution for application migration, either planned or unplanned. See also “nodeless resource group.”

resources Objects that AutoStart manages inside an AutoStart domain. Resources include: processes, services, managed IP addresses, node aliases, data sources. Resources that are components of an application are managed as part of a resource group or module instance. You can also define resources that you define to AutoStart so you have the convenience of starting and stopping them from the AutoStart console.

response test An optional test of a process’ response state. Response monitors test that the target process is healthy and responding correctly. This test is optional, but indicates a processes’ health. AutoStart provides the option of user-defined response monitors for processes and services which can provide a more detailed and accurate status of the process or service.

response time The time period within which the agent should expect to receive a message from an aware process.

rule A user-definable Perl script which is evaluated when a trigger condition changes. Rules implement AutoStart’s monitoring and management capabilities. Rules perform both system-defined and user-defined actions in response to events within an AutoStart environment.

rule APIs A collection of AutoStart subroutines that are accessible from a rule for performing AutoStart-related management capabilities. Most rule API subroutines can also be used in resource group scripts.

rule interpreter The agent process that evaluates custom rules (and resource groups) on primary agent nodes. For more information, see “The rule interpreter” on page 138.

S

scope The range over which a trigger applies. For node-sensors, the scope indicates for which nodes the trigger applies, for process sensors, the scope allows for the specification of the processes for which the trigger applies.

secondary agent Two AutoStart processes, the agent and process monitor, that provide management capabilities of a node. It does not execute rules, nor does it maintain a copy of the replicated database. AutoStart domains can have as many as 5 primary nodes and 5 secondary nodes, depending on your AutoStart configuration and network conditions. See also “primary agent,” “agent.”

security The list of users with permissions in the domain and the level of access allowed to each user. AutoStart provides three forms of user-level access: User, Operator, and Administrator. For detailed information, see “User access and security” on page 32.

sensor A piece of data to be monitored. Provides data values for use by triggers and rules. Sensors are published by the agent, by proxy processes, and by any other AutoStart aware processes. The following resources, when managed by AutoStart, each have one built-in sensor that can be used to monitor its state: process or service, node, data source, IP address, NIC, resource group.


Glossary

shutdown sequence The list of resource objects that will be taken offline, the scripts that will execute, the events that will be posted, and the delays that will executed in the order specified when attempting to take the resource group offline.

Standby NIC An AutoStart usage for a NIC that designates the NIC as the destination for network card failover. A Standby NIC is not currently in use. AutoStart will not assign managed IP addresses to the NIC except during NIC-to-NIC failover. A Standby NIC is used only to receive IP addresses during NIC-to-NIC failover. Standby NICs are not used for general IP address assignment.

start script A script used in conjunction with a managed process which performs specific actions each time the process is started. The start script is often used to launch a number of related processes which are necessary for the proper execution of an application. See also “stop script.”

startup sequence The list of resource objects that will be brought online, the scripts that will execute, the events that will be posted, and the delays that will executed in the order specified when attempting to bring the resource group online.

state monitor AutoStart provides state monitoring capabilities for processes and services defined in the domain. There are two types of monitors to test different aspects of a process’ health—the response monitor and the existence monitor. See also “response test,” “existence test.”

states An indication of the current condition of an AutoStart resource. State icons display in the AutoStart console. For descriptions of resource states, see AutoStart online help or the EMC AutoStart Administrator’s Guide.

stop script A script used in conjunction with a managed process which performs specific actions each time the process is shutdown gracefully by AutoStart. The shutdown script may ensure that all processes associated with an application are properly closed. See also “start script.”

subnet mask A TCP/IP configuration parameter that extracts network and host configuration from an IP address. This 32-bit value allows TCP/IP to distinguish the network ID portion of the IP address from the host ID portion. The host ID identifies individual computers on the network. TCP/IP hosts use the subnet mask to determine whether a destination host is located on a local or a remote network. A subnet mask is a 32-bit number expressed as four decimal numbers from 0 to 255 separated by periods, for example: 255.255.0.0.

take offline Initiate an attempt to execute a resource group’s shutdown sequence. When the shutdown sequence successfully completes, the resource group is in the Offline state.

T

trigger An object that monitors a sensor for a logic condition and reports any matching conditions to one or more rules. Triggers are the objects that cause rules to execute. There are four types of triggers: sensor-based (which monitors sensor values and fires when it recognizes a specified sensor value), on-demand (which fires when a user or rule tells it to), scheduled (which fires at a specified time), and event log (which fires when a specified event is written to an AutoStart event log).

unaware process A process that does not use the AutoStart runtime API and so is unaware of AutoStart. This includes third-party and shrink-wrapped software that is monitored by AutoStart. Unaware processes can be associated with a proxy process to allow AutoStart to monitor various conditions. Compare to “aware process.”


Glossary

U

unplumbed NIC A network interface card (NIC) on which no IP addresses, including no base IP address, reside. An unplumbed NIC cannot be managed by AutoStart. A NIC can become unplumbed if AutoStart fails over its addresses to another NIC on the node. If a NIC becomes unplumbed but becomes available again, you can reset it in AutoStart. Note that on Solaris nodes unplumbed NICs are tested, so there is no need to reset a unplumbed NIC on a Solaris node.

Usable NIC An AutoStart usage for a NIC that identifies the NIC as available for IP assignment. A Usable NIC can be assigned IP addresses, and may be used if its status is Alive at the time of IP address assignment.

utility process A process that can be executed from AutoStart through a rule or from the AutoStart console. The utility process is started, but not managed, by AutoStart. The utility process is typically a short-lived program, often used to perform some auxiliary function. Utility processes are also referred to as UtilProcs.

V

variable See “persistent variable.”

virtual IP address The movable IP address that is assigned to a Network Interface Card (NIC) by AutoStart. See also “physical IP address,” “NIC (network interface card).”


Glossary


INDEX

Aactuators 129, 143, 152, 187admin-level security 26agents 12, 30, 187

quick reference guide 175rules and 152

AIX file system data source 86AIX LVM Volume Group data source 83APIs 151, 154, 193attaching a data source 57 to 58

parallel attach 58 to 61attributes 187AutoStart 5.2

SQL 2000 module 101AutoStart console 166 to 168, 187AutoStart database 31, 151, 192AutoStart domain 8 to 28, 188

in the console 10multiple 10name format 10quick reference guide 174rules that monitor 128

AutoStart-supplied sensors 142aware application class 160aware processes 120, 142, 187

Bbackbone 187base IP addresses 40, 42, 43, 191built-in sensors 120

Ccommand-line interface (CLI) 168 to 171, 188Composite data source 63 to 65, 188

used with Exchange 2003 104configurations 124 to 125console 166 to 168, 187

Ddaemon processes 121data source checkpoint 59, 188data sources 56 to 86, 188

attaching 57 to 58licenses for 168list of 56parallel attach 58 to 61quick reference guide 176

database 31, 151, 192debugging 147, 154definition files 28delay 188demo 150device groups 78

disable a rule 188disable monitoring 188DNS solution 43domain network 188

EEMC licenses 168EMC Mirroring for Windows data source 65 to 67, 189

used with Exchange 2003 104, 105 to 106used with Oracle on Windows 112 to 113

EMC MirrorView Mirroring data source 67 to 70used with Exchange 2003 104

EMC SRDF Mirroring data source 70 to 79used with Exchange 2003 104, 107 to 108used with Oracle on UNIX 117 to 118used with Oracle on Windows 114 to 116

enable monitoring 189event-log triggers 134, 135 to 136Exchange 2003 module 103 to 108

node limit 91existence monitors 52 to 54, 57, 189ext2 file system data source 86ext3 file system data source 86

Ffailback 11, 187failover 11, 189

across node groups 92NIC-to-NIC 36

failure detection 12, 189examples 13 to 17

fire a trigger 189firewalls 12ft_DataSourceState 139ft_EventSensor sensor 135ft_IP_AddressState 139ft_NicState 139ft_NodeState sensor 139ft_ProcState 120, 189ft_ProcState sensor 139ft_RG_State 139

Gglobal variables 160 to 163, 189

Hheartbeats 12, 13, 189help 167heterogeneous environment 189HFS file system data source 86hpfs file system data source 86HP-UX file system data source 86


Index

HP-UX LVM data source 84

II/O redirection 190IIS 88IIS module 109 to 110IP addresses 40 to 43

base 40, 42, 43, 191for isolation detection 20limit 42MAC address 42managed 40, 42, 43, 190physical 191quick reference guide 177removing 42versus name service entry 41virtual 195

isolation detection 190isolation network 19isolation script 21, 190

JJFS file system data source 86

Llicense 168Linux file system data source 86log files

trigger that writes to 135

MMAC address 42, 190managed IP addresses 40, 42, 43, 190managed process or service 190managed processes 120managed resources 190Microsoft Exchange 2003 module 103 to 108Microsoft IIS 88Microsoft IIS module 109 to 110Microsoft SQL Server 2005 module 99 to 102Mirroring for Windows data source 65 to 67, 189

used with Exchange 2003 104, 105 to 106used with Oracle on Windows 112 to 113

MirrorView Mirroring data source 67 to 70used with Exchange 2003 104

MirrorView/A 68MirrorView/S 67module instances 191modules 98 to 118, 190

licenses for 168node limits 91quick reference guide 178

monitors 48 to 54, 57existence monitors 52 to 54response monitors 50 to 52user-defined versus built-in 48 to 49

Mount Points data source 79 to 80

Nname format for domains 10name service entry 41NetBIOS 43Network File System (NFS) data source 85network interface card (NIC) 31 to 38, 191

failure 41network partitioning 12networks 9 to 10, 11, 13, 188

domain 17 to 18isolation 19 to 22Symmetrix 23verification 22

NFS file system data source 85NIC groups 32, 38 to 40, 191

quick reference guide 176NICs 31 to 38, 191

failure 41quick reference guide 176

NIC-to-NIC failover 191node aliases 43 to 46, 191

quick reference guide 177node group separator 90, 92node groups 92 to 93node proxies 46, 120, 142, 191

quick reference guide 183node states 149, 191nodeless resource group 89, 191

managing with rules 128nodes 30, 191

node limit by module 91preferred nodes list 89 to 93quick reference guide 175using rules to determine 91

Oon-demand triggers 134, 141online help 167operator-level security 27Oracle for UNIX module 116 to 118

node limit 91Oracle for Windows module 112 to 116

node limit 91output from rules 154

Pparallel attach 58 to 61, 191Perl scripts 128, 130, 151persistent variables 191personality swap 77point-to-point communication 18preferred nodes 191preferred nodes list 89 to 93primary agents 12, 30, 192Print Services module 110 to 112process proxies 120, 123, 142

configurations 124 to 125quick reference guide 182

process states 192


IndexIndex

processes 120, 121 to 123configurations 124 to 125quick reference guide 180

proxiesnode 46

proxy processes 120, 192publish sensors 192

RRDF data source 70Registry checkpoint 192reiser file system data source 86relocate 192replicated AutoStart database 192RepliStor 5.x data source 62RepliStor 6.x data source 61 to 62require ft.pl 21Reserved NICs 38, 192resource groups 88 to 96, 193

managing with rules 128quick reference guide 179

resource states 149resources 193response monitors 50 to 52, 57, 193response time 193rule interpreter 132, 193rules 128 to 132, 193

demo 150firing triggers 148output 154Perl scripts 151planning 144 to 147quick reference guide 184running 148tutorial for 150writing 149 to??

SSANs 80scheduled triggers 134, 136, 145scope 193script

isolation 21scripts

Perl 130, 151secondary agents 12, 31, 193security 26 to 28, 193sensor-based triggers 134, 137 to 141, 145sensors 129, 141 to 143, 146, 193

AutoStart supplied 142for processes and services 120non-state sensors 140publishing 192sensor classes 142sensor-based triggers 137 to 141state sensors 139user-define 142

services 120, 125quick reference guide 181

shared disk configurationused with Oracle on Windows 117

Shared Disk Device for Windows data source 58, 80 to 81used with Exchange 2003 104, 106 to??, 106 to 107used with Oracle on Windows 113 to 114

shutdown sequence 94 to 96, 194Solaris file system data source 86split brain 12SQL Server 2000 module 98SQL Server 2005 module 99 to 102

node limit 91SRDF Mirroring data source 12

used with Exchange 2003 104, 107 to 108used with Oracle on UNIX 117 to 118used with Oracle on Windows 114 to 116

SRDF/Asynchronous (SRDF/A) 71, 75SRDF/Synchronous (SRDF/S) 71, 73standard processes 120Standby NICs 38, 194start script 194startup sequence 94 to 96, 194state monitors 48 to 54, 57, 194

existence monitors 52 to 54quick reference guide 179response monitors 50 to 52user-defined versus built-in 48 to 49

states 149, 167, 191, 192, 194stop script 194subnet mask 194subroutines 151, 154, 193Sun Solstice DiskSuite (SDS) data source 85Symmetrix 12, 23, 70

Ttake offline 194Telnet processes 126tracing 154

quick reference guide 185triggers 129, 130 to 132, 133 to 141, 145 to 146, 152 to 153,

194fire 189firing 148quick reference guide 184

tutorial for writing rule scripts 150

UUFS file system data source 86unaware processes 120, 194UNIX daemon processes 121UNIX file system data source 58, 86unplumbed NICs 32, 39, 195Usable NICs 38, 195user access 26 to 28user-defined sensors 142user-level security 27utility processes 120, 126, 195

quick reference guide 182


Index

Vvariables, global 160 to 163verification network 22VERITAS Volume Manager (VxVM) for Windows data source 81VERITAS Volume Manager for Solaris data source 81virtual IP address 195VxFS file system data source 86

WWANs 12, 18, 90, 92Windows Network Share data source 58, 82Windows Print Services module 110 to 112Windows Registry Monitor mirroring utility 24 to 25Windows Registry Monitoring mirroring utility 192WPS module 110 to 112

node limit 91


hk.emc.com · EMC AutoStart Concepts Guide 1 CONTENTS Chapter 1 AutoStart Domains EMC...

Documents

Transcript of hk.emc.com · EMC AutoStart Concepts Guide 1 CONTENTS Chapter 1 AutoStart Domains EMC...