FilterfreshFilterfresh

Fault-tolerant Java Servers Through Active Replication

Arash BaratlooArash Baratloowww.cs.nyu.edu/phd_students/baratloowww.cs.nyu.edu/phd_students/baratloo

• Investigation of failure models in distributed Java applications

• Provide transparent fault-masking (to users and to programmers)

• Support highly available services in presence of failures

• Remove single-points of failure

FilterfreshFilterfresh

Remote Method Invocation (RMI)100% Java, hot, new, easy-to-use

and

Reliable Object Services (ROS) Interest in Providing:– support active-active replication– support Java objects

Motivating FactorsMotivating Factors

RoadmapRoadmap

Motivation

– RMI Registry & crash failures– RMI Server Architecture & crash failures– A Unified Solution -- process group

approach– Fault-tolerant Registry– Fault-tolerant RMI– Conclusion

RMI in a NutshellRMI in a Nutshell

• Servers register with the local registry

• Clients looks up a server at a well known registry

• Given a remote reference, client performs a remote method invocation

registry

client

serverlook

up"a

bc"

bind to

"abc"

Limitations of RMI RegistryLimitations of RMI Registry

• The “well known registry” requirement too restrictive for failure recovery

• Single point of failure• Can not support replicated servers, thus,

highly available servers

FT Registry requires...FT Registry requires...

• Distribute and replicate registry servers• Replication strategy to maintain a consistent

state• Failure detection and removal of failed

registry servers• Failed objects must be restarted

automatically• Dynamic addition of registry servers

clientapplication

server stub

Transport layer

serverapplication

server skeleton

remote referencelayer (RRL)

remote referencelayer (RRL)

RMI ArchitectureRMI Architecture

• RRL assumes a stream-oriented transport• Transport layer implemented on TCP/IP

Architecture Architecture (cont…)(cont…)

client

stub

RRLtransport

RRL

skel

server

interface Server public void foo();

class Client { ... Server s = lookup... s.foo();

Architecture Architecture (cont…)(cont…)

client

stub

RRLtransport

RRL

skel

server

interface Server public void foo();

class ServerImplextends ... { public void foo() { ... }

Architecture Architecture (cont…)(cont…)

client

stub

RRLtransport

RRL

skel

server

client

stub

RRLtransport

RRL

skel

server

class ServerImpl extends

UnicastRemoteObject { public void foo() { ... }

Architecture Architecture (cont…)(cont…)

Transparent FT system implies RRL or below

FT Servers Require...FT Servers Require...

• Distribute and replicate servers• Replication strategy to maintain a consistent

state• Failure detection and removal of failed

registry servers• Dynamic addition of registry servers• Object reference must remain valid after the

associated object has failed

A Unified Solution...A Unified Solution...

Process Group Approach where all non-faulty

objects– form a group– consistent view of the group– interact through reliable group primitives --

all or nothing– total order on group primitives

FortunatelyFortunately

Process Group Membership is– well understood problem and protocols– well tested (ISIS, Transis, Amoeba, etc.)– basis for virtual synchrony

Equivalent Problems* (implement one, get all)– Group Membership– Reliable Failure Detectors– Reliable and ordered multicast

* Chandra and Toueg. Unreliable failure detectors for Reliable Distributed Systems. JACM, March 96.

UnfortunatelyUnfortunately

Process Group Membership is– as hard as distributed consensus– impossible in purely asynchronous systems with

crash failures*

Our solution– the standard “timeout” assumption– variation of protocol used in Amoeba OS**

* Chandra, Toueg, Hadzilacos and Charron-Bost. Impossibility of Group Membership in Asynchronous Systems.

** Oey, Langendoen and Bal. Comparing Kernel-level and User-level Communication protocols on Amoeba. ICDCS 95.

What We Provide...What We Provide...

A Group Manager Class– 100% Java– build on top of UDP/IP

Implements– group creation– join operation (with state transfer)– leave operation– failure detection and recovery– reliable multicast

All events are atomic and totally ordered

Multicast PerformanceMulticast Performance

• Pentium Pro 200, Linux RedHat 4.0, Fast Ethernet hub

0

10

20

30

40

50

60

70

Tim

e (

ms

ec

)

1 byte 512 1024

Message size (bytes)

local RMI

remote RMI

multicast-1

multicast-2

multicast-4

multicast-8

FT Registry ArchitectureFT Registry Architecture

• registry on each host/domain

• group managers ensure reliable ordered events

• support dynamic joins w/state transfer

ft registry

rmi registry

group mgr

ft registry

rmi registry

group mgr

server

bind

multicast

FT Registry Architecture FT Registry Architecture (cont…)(cont…)

• lookup becomes a local operation

• detect and remove failed objects

• consistent global state

ft registry

rmi registry

group mgr

ft registry

rmi registry

group mgr

client

look

up

FT Registry PerformanceFT Registry Performance

• Pentium Pro 200, Linux RedHat 4.0, Fast Ethernet, Ethernet hub

0

10

20

30

40

50

60

70

80

bind lookup

RMI Registry local

RMI Registry Remote

FT Registry-1

FT Registry-2

FT Registry-4

FT Registry-8

RMI & FT RegistryRMI & FT Registry

• support multiple servers register with a same name

FT Registry

clientstub

RRL

transport

RRL

skelserver

transport

RRL

skelserver

RRL

skelserver

• can now support

recovery from server failure

What if...What if...

In the event of server failure...

FT Registry

clientstub

RRL

transport

RRL

skelserver

transport

RRL

skelserver

RRL

skelserver

Ouch!

Failure RecoveryFailure Recovery

• The old connection is patched with

a connection to a non-faulty server• Illusion of a valid object reference• Transparent!

FT Registry

clientstub

RRL

transport

RRL

skelserver

RRL

skelserver

"reverse" lookup

transport

• A “reverse”

lookup returns

a name given a

wire connection

Failure Recovery Failure Recovery PerformancePerformance

?Working but measurements have not been

made

FT Server ArchitectureFT Server Architecture

• Client has the illusion of a single server• In reality, we have active replicated servers• Highly available?

serverskel

RRL

groupmgr

serverskel

RRL

groupmgr

serverskel

RRL

groupmgr

serverskel

RRL

groupmgr

clientstub

RRL

transporttransport

Highly Available ServersHighly Available Servers

• Group managers ensure reliable ordering of events across all servers

• Guarantees servers have a consistent state• Failure detection and removal of failed

servers• Dynamic addition of servers w/state transfer• Illusion of a valid server reference even after

the associated object has failed

ConclusionsConclusions