'6Zy[ytVH] ur pasodord rreaq e^eq slocoloJd Eur>1co1q

-uo1q 'drilqco?quou paIIBc arE paqceal aq uec (llultuocJo 1:oqe 'E'a) uorstcap llturuoc luaJslsuoa E eJoJaq allsp"1pg e ;o ,(r6aoca.r aql l1lun lre.l,r lsnlu sa11s leuorle.radoeJaqra elrels e ur Bulssacord uollcesue{l allzal Jalau.leqlslocolord llurruoC 'peseelal aq u€a uollcesuEJl eql '{qparrnba.l s{col aql luq?, os (l.zessaaau JI) uollcesue:1 aq1

lroqE oJ alIS IEuoIJEJado aq1 roJ alqeJaJard sr 1r 'pEalsul'paJelocal sEq e11s palreJ E lrlun uorlcesuerl e 1o sse:E

-o.rd eq1 1co1q dlaliugepul oJ pacJoJ aq 01 salls Iuuorl-erado :o3 elqEJeiolul sr 1I suorlEcrlddu '{uetu .Iog

'(loz,lto.ts 'qIBf,f,)s'6ZONI'I'9Zdi{v'I'6l,li[]i[YH'6aAYUC'll,I"i'If, '91'gsrY])p-.=*oitia

".Eq "i"q sfotolora 1iu,u:o' 1€ra^aS 's1oco1otd

V-*rz;o, peII€c aJE lltcru:o1e uorlc€su€J1 SurarasardJOJ SIocoJoJd 'asEq E?Bp ?ualslsuocuI uE ur sllnsaJuorslcep pexrul Y liululoc .(lsnorurueun Jo lJoqE dlsnour-rueun Jaqlre salrs aq1 1eq1 Eurrnsur Jo auo sauroceqrualqo-rd aql 'Ia^aI 1eco1 aq1 1e dlcruolu sapraord qcrqlrd8alerls l:arrocar lecol E seq aJIS qcEa ?Bql Eununssy'pa,llo,\ur s1 aJrs auo ueql eJolu uaqu papunoduroc sr

.(lrcrurole uo1lcEsuer1 Fulaalueren8 1o rualqo:d aql'aIqIsJaaaJJI ar€ lJoqE puE lllu

-ruoc qlog 'uoIlcBSuBJl eql ?JoqE III,tr alls, aq1 d"raaocaruodn ilalerpaurur ueql 'paqo€al si lurod Jrurluoc aq1

aJoJaq sJncco aJnlr€J e 11 '1srs.rad sllnsaJ -s1.!

Jo auou 1€qlos uollcEsueJl eql ,rto qcnq 01 aaluu:enE leuolllpuocunue sqJoqo ue 'dpepurg 'saJnlluJ a1dr11nu Jo lualaaql ur uale 'uorlsldruoc 01 uorlc€sueJl eql alncexa 01

ealue.renE lEuolllpuocun ue st f.u.uutoc Y 'uorlcEsueJl aql

lraqn o1 Jo J?'-tuutoc 01 saprcep alIS eq1 aJaqll peqcEeJ Si

V4ud y.ttttttto, E 'uollncexa s1i Surrnp aturl-aulos 1Y 's'^'oI

-ioJ s€ patreu sI uorlcesuEJl a18urs e 1o Surssacord aq;'Ior,rvul '6e[NI']l rualqo:d poolsrepun lIem' e sI esBc

#s--"faur= aq1 ur-.(lrcrurole uorlcEsuurl Eura:asa'r4

'1uau:aEeuetu uollcESuEJl luerlrseJ1o slcadse aql Jo aulos ssncslp r\dou eil 'rualqord lIncIUIp.dr"r,

" sr '>Uo^rleu suollEclunluruoc eql ;o 3utuot11-red pue saJnlIEJ ays d:e"ryqre Eulpnlcut 'saJnIr€J snorJel01 luallrsaJ erE 1Eq1 luau:aBeueru uollcesuEJl JoJ sloco1

-ord SuiuErsaC 'tuiqf Jo euou lE Jo sells 11e 1e passacordeq lsntu 1r:u6r1e:ado alurole ({11ecrEo1) e sr u:a1sds aseq

-ElEp palnq-lJ1srp e $o uu!4cttsur1,.L1 e uorl-rugap '{gPuno.r?4aug '2

'["iAUUyS] u! pessncslp aJE lnq '"radudsrql 3o adocs aq1 aplslno aJB suolsuelxa asaqJ 'palJesqo

"riri,.ng 3o adll aq1 ur ,(lurelracun 01 pue 'ssol a8Bssaur,{.rerlq"re 'suor11:ed >IJoi!1au ol luerllsal apeLLI aq uec

locolord pasodo:d eql 'sarnIIEJ alrs o1 uol]lppu uI'uorsslulsueJla: 1e s1drua11z lnlsseccnsun Eur

-Arasqo .(q pue slnoaurq .(q sarnpel a1ls I;t:arr pue 1ca1epuEc }IJor\'rlau eq1 1Eq1 alunssE osIE art '(luas JepJo aq1

ur pallecal "q

s.Erss".u 1eq1 a-rrnbar lou op e,tr. ':aaalr'oq)

sgsl 186r o qz'00$2u00 I 0000 I 18 I 6-z89rHC

'gIoo-c{l'-6GoooN 10Br1

-uoC punuuoC map(5 6cruo4aelg [p EN at{l Pve 'q7Z1A-SL1ZCWAlwrc aorgo qoiBasau Iury S n eql '06q8-gir luBrC qcEaEaU )glluala$go airgg aarbg 4y S'O eq1 {q palosuods ssr qorEasal slt{J

saJrs leuorlerado o,\4 ,(ue uaa,nlaq suollPcrunulruocluiod-o1-1urod saprrrord liJo$'lau suorlecrumuruoc Eut

-Ilrapun aql 1Eq1 arunss€ aa,r ':aded aq1 lnoq8norql'1oco1o.rd pasodord aql Jo salnq

-rJl1E aql sazlretuulns uollces lsPI puE qlJU aqJ 'ssau

-lcaJJoc;o;oord JaIJq E qclais osle all 'a]E]s Jualsrsuoc€ ur uorlcEsueJl aql al€ulruJal 01 paaluEJen8 sr 1oco1o.rdaq1 'IeuorleJado surBural ays ai8urs u se Suoi sY 'saJnIIeJ

ays f.re.r1tq.ie o? ?uaTIISaJ sI qrlqa locolord uollEulluJa?pazlleJluecap e luasa:d a.!r 'uorlcas qlJnoJ eql uI 'ioc

-olo:d paz1leJ1uacap e 1o eldurexe pooB e sr 1t 'dlpuocag's1oco1o:d uoll€uIruJel Jo Eurpuelsrapun ue pJentol

Ierluassa st 1oco1o.rd lltuuroc 3o Eurpuelsrapun uE '1sJIJ'suosEeJ ori,!] JoJ papnlcur sr aldruexa aq; '1oco1o"rd Tru-ruoa Eurlcolquou E 1o eldruexa ue acnpoJlur ei uorlc3spJrql aql uI 's1oco1o.td pazlleJluecep pue uorlpuIrxJelrritt.,rtip pue Eurug"p sepnlcul qaltlr\d lerralEru puno'r8-1ceq dressaceu aq1 dolaaap afi. 'uo1Jcas puoces aql uI'snoIIoJ se paztue8ro sr radBd 3q1 Jo JapurEueJ eqJ

'salrs eq1 Euorue lsrxa sdrqs-uorl€lal artuls/:a1seu: aJeq.& slocolord paz11e:1uac re1

-ndod a.roru aql qll,t,r palseJluoc eq uec sIqJ 'aloJ crJlarrr-urIs pue lenba ue sarunssE ells qc€e '1oao1ord pazrleJl-uecep e uI 'locoloJd pazl7o.tyuacap : Jo aldluexa ue sr

1oco1o:d aqJ 'sseulsarJoc sJr 3o Soord P q?la{s puE locol-o"rd uolleuguuel qcns auo asodo:d a.r,r raded s1q1 ul

'salrs I€uollEJado 11e 1e salepdn aq1 Eurllelsur ({1qe-raaoce.r) dq :o sals Eulledlcrped IIE 1€ uollcEsueJl aql

1no Euolceq "raqlra dq alels luelslsuoc e 1 aseqelpp aqleloru 01 sr 1oco1o.rd uolleulruJel e 3o leo8 eqJ 'pslolulsg 1oco1o:d t lrptr.Luual E 'alBls aJESun (d11er1ua1od) e utuorlcesuBJl pelnqlJlslp u 3o Eurssaaord aq1 aaeal saJnIlEJ

elrs Jelaueq lL 'uollcosuoJl E sl ruelsds eseqelEp pelnq--rJlslp E UI {JO.^ JO lrun alqEJaAOCeJ lsallPurs eqJ

uollcnpo4ul 'I

'{ro,r,r}au aq1 uorlyed lou op }€q1saJniI€J alls Jo suolleurqruoc 1l€ 01 luarllsa-r sr 1oco1o'rdpasodo:d aq; 'aloJ slJlaluru,(s pue lenba ue saurnsseilrs qren aJaq^r 'locolotd pazTyolyuacap e 1o alduruxaue sr 1oco1o:d aq; 'ssaulcaJJoc sy 1o ;oord e qcla>Ispue 1oio1o.rd uorleurruJal qcns euo asodord au.Ieded s1q1

uI 'suollcESuEJl aJnlnJ q1llr. paacord 01 salls leuorle:adoEurlqeua alels aJ€s € 01 esEqelep aq1 eJolsal 01 pe{olursr

-1oco1ord 7to!4zltr1;u,q e 'alels eJesun (Ip11ua1od)

E ur uollcesupJl pelnqrJlsrp e Jo Eulssaco.td aq1 elealsaJnIIEJ a1ls Jeleueqit ':uallcrsu:Dl1 e sr u:a1sds aseqElEppelnqiJlsrp E ur lJonr Jo llun elqEJa^oJer lsaliEtus eqJ

L7

lcs4tqY

otut ol'r,7o3' fi,a7atlta gnt u"t o tV7o3 I o fl1zs,.ta rnun

yuattt4lndag gSgguu.Ls!a!!O a cua?cs,t a Tndut oi

uaaqs apo

.loco]ord uolleulurlatr Pezlle4uacao Y

SKEI1Uib.l. in Section 3 we review a nonbLocking commitprotocol and its ProPerties.

Termination ProtocolsTerm'i,nation protocols are used in conjunction with

nonbiocking commit protocols. A termination protocol is

invoked when occurrences of site failures render theLontinued execution of the commit protocol impossible'The purpose of lhe termination protocol is to identify the

of"Juri""ut sites and move them toward a commit deci-sion which is consistent with both operational sites and

lailed sites. It is the responsibility of a nonblocking com-

rriit protocoL to always leave transaction processing in a

state such thal the termination protocol can proceed'ifr" -u1o. contribution of this paper is the terminaLionprotocol presented in Section 4.

Decentralized ProtocolsIn a (completely) decentralized protocol' as the

..-" ,rggL"ts,-theri is no hierarchical ordering of the

sites. Iniilad, each siLe communicates with every othersite, and each site assurnes a symmetric role'

A decentralized protocol consists of successiverourrds of message interchanges where every operationalsite participaLesln every round iYlttrln .a

single round' asite sends identicai -"jtug". to al} of the other partici-pating sites, and then waits to receive a message fromL."ft"of Lhem. Of course, a site may fail while sending itsmessages during a round and only send Lo a subset of itsintended receivers.

A very simple example of a decenlralized protocol isL}re simpl'e decetttraltzed. conuni't Strotocol which is thedecentrilized anaiog of the centralized two-phase com-mit protocol. Assuming that a transaction has been sentto eich site for pro"esilng, the protocol consists of a sin-

gi" -".uug" round where eachiile sends its vote ("yes"

[o commit] "no" to abort) to aII of Lhe other sites AJtera site has coLlected voLes from aLi of the other sites' itwill commit only if all votes were "yes" Like the two-phase commit protocol, this protocol is functionalLycorrect but nol verY robust.

Decentralized. protocols require n(n - 7) point-to-point messages during a round, where n is the number of

iarticipantsl If a bro-adcast faciiity is present' then thisi"d."". to n broadcast messages' Therefore' decentral-ized protocols are attractive only in networks where.n"r=.g", are cheap or a broadcast facility is available'Fortun"ately, one or both of these condit'ions are likely tobe true in a high speed local area network (e g' ETHER-

NET [METC?6]). Because of their inherent symmetry'deceitralized protocols tend to be easier to understandand Lo implement than centralized protocols'

3. A Nonblocking Decentralized Commit Protocol'We illustrated a simple commit protoeol in the pre-

vious section. Unfortunately, it is not a very robust pro-tocol: it often blocks the progress of a transaction whensites fail. lTe now present a nonbloeking commit proto-col. In addition to serving as another, more complexexample of a decentralized protocol' it wiil aiso intro-duce the common properties of aII nonblocking proto-cols, These properties are used in the design of termina-Lion proLoeols.

The nonblocking deeentralized eommit protocol was

fust introduced in [SKEE81b].

The ProtocolThe nonblocking protocol is derived from the simple

protocol by adding another message round and delayingihe commit point of a transaction until the end of thesecond round.

1n the simple commit protocol, a site would commilat the end of Lhe single message round if all sites hadvoted gres. In the nonblocking version of the protocol, an

all yes vote would trigger a second round of messages,where each site sends prepared to comntii messages andwaits. Upon receiving prepwed to com'rrfi't messagesfrom all of its cohorts, a site will then commit the tran-saction. (The proLocol is given in its entirety in Figure1.)

.Whenever a siLe detects the failure of another sitewhile executing the eommit protocol, it will invoke a ter-mination protocol. The detection of the failure and thesubsequent invocation can occur durlng either messageround.

Properties of Nonblocking Commit Protocolsln the nonblocking decentralized commit protocol,

we can identify flve distinct states in processing a tran-sacLion. Briefly, they are: an initial state where the siteis waiting to receive the transacLtoni a u-ruLt state wherethe site has voted "yes" and is waiting for all of the othervotes; a prapared sLaLe where the site has sent "preparedto commit" messages and is waiting for a similar mes-sage from aII cohorts; and two flnal states, oborf andcontrnit.

The transactlon staLes of any commit protocoi canbe partitioned into two sets: committable and noncom-mit[able. A state is called corrrrtu.-i.ttable if occupancy ofthat state by any site implies that all sites have voted"yes" on committing the transaction. A state that is nota committable state is called ttottcornnvi,ttahle.l In thenonblocking commit protocol presented above, both the

Initial Phase. Transaction is sent to all sites'

First Round. Each site broadcasts its vote, ges ot rLo,

for the transaction.If a site receives all gtes votes during this round,lhen a second round is initiated. Otherwise, thesite aborts lhe transacLion.

Second Round. Each siLe broadcasts a prepared to cont'rzd.f message.

Upon receiving a prePured.., message from eaehof its iohorLs, a site commits the transaction'

Figure 1. The nonblocking decentralized commit proto-col.

lTo call nonconmittoblesince a trmsaction that is notst;1] be aborted,

vtates @borta.bls rould be misleadirg,in a final commit state at anY site cm

28

'PapPB am sPmoJ atBssau IB

-uol1rppB ssalun pelcalaPm oB suosraap lualslsuocu1 ar{l raJoular{llnJ'efalduocul sr salE lauorlErado Jo mlBls aql ?uluJacuoo uonauJoJ-u aq1 puP rncco samIEJ a1B IBUorfrPpB qloq ual{a suo6lceP lualsrSuoo-ur a>{Eu saurlauos slocoloJd Punol aIEUF lBr{1 salB4snlll s-rqJz

aql Jaq1la sardncco 11 Jr aSESsaru a'lqnJnul'ltr,ocuau epuas III./II lr pue :alels lrluuroc aq1 Jo alpls paJedaJd aqlJaq1la serdncco ,{lluaJJnc 1r Jr aEpsselu a'lqnlJ?1"u1uac epuas IIin y :a1€ls 1:oqe aql saldncco ,(11uar:no 1t ;i aBes-saru J"r,oQo uE puas IIr& ells Y '8 uollcas uI paqlJcsap loc-o1o:d lruuroc pazll€Jluacap Eurlcoiquou aqJ uror; 1t Eur-Iolur Japrsuoc '1oco1o:d aq1 Eutsn 3o aldurexa ue sy

'1r JJoqe 'asr-^ Jaq1o :uoIl-cesupJl aql llruruoc uaql 'pe^Iaca.r sr aBessauralqnlf.Llt'u"Loc auo lspal 1e JI 'a1n5 lIurEoJ a1du1g

:a1nr 8ur,to11ogaq1 o1 Sutprocce aIEJS IEuU E oJ.,(11aartp alolu IIrr\r alrs E

'sluedrcrped aql II€ uroJJ saEessaut Euuraca: uod11

'e1B1s

JJoqe ue Jou alEls e1qE11ltuuloc B raql-Tau sr a1Els uol1c€suEJJ aql Jr a'lqnlfluturocuou

puE 'alE1s alqElllul-uroJ e sr a1els uollsesueJ? aql Jr alqolr,1u1uot

'a1ElslJoqE IEUU e sr alels uollcesuEJl aql JI +Joqn

:saEessaur alqtssod aa:q1eJp aJaqJ 'a1EJS uolJcpsuEJl JuaJJnc s1t,{q d1a1os pautur-Jalep s1 aTs e ,(q luas a8ussaru eqJ 'punoJ srql Eutrng'saEessaru Jo punoJ alEurs e Jo s'lsrsuoc 1oco1o"ld aq;

IocoloJd uollsulul.lal. aldu(s Y

'squ,ndt,ttqr,nd aq1 se dldrurs salls lEuorlErado o1 ra3a.r oslu1ll,\{ ar{d 'punoJ e Eutrnp saalasuaql ol saEessaru luassaqrs qEnoql s€ {eads IILII eI\ 'uorlelou dSttdurs o1

'selrs I€uol1e:ado 3o JaqtunuIBrlrur aq? o1 lenba sI spunoJ Jo Jaqrunu urnlulxBtu aI{J'spunoJ leuorlrppe asnec .(Bu: 1oco1o:d aql Jo uorlnraxaaq1 Eur.rnp saJnIrEJ alls IEuoIlIppE 'Jale,t!.oq :saEu€qcJaJuIaEBssaru Jo spunoJ o,tr1 arrnbat 11i.rut 1oco1o:d uoIJEuIluJalluallrser aq1 l11eru.ro1g 'Iro,lrJeu aq1 uor1t1:ed lou op 1Eq1saJnIrEJ aJIs Jo suoilEurqluoc II€ 01 JuarIIseJ sI 1€I{J locol-o:d aldrurs eq1 Jo uo{suelxa ue luaseJd uaql IIy( e,11

'lo coJ-o.rd uorleunuJal pazlleJJuacep E uI pasn sBepr cISEq eq1acnpoJlul 01 a^ras 1ll,r slqJ 'uorlncexe s1i Sur.rnp sarnIrEJalrs JaqunJ 01 luaIIISar lou sI lEql loJoloJd uorleu-rruJal pazrlEJ1uacap a7dtuts e luasa:d llrrr,r ana 'lsJld

'punoJ puocas aql IrlunJr laelap Jou II1,{r sraqlo alrqu 1oco1o:d pazqe.rluacap Eur-ricolquou eqJ Jo auo punoJ uI ernlleJ E lcaJap r(eru saTsaruos 'aldtuexe Jod '1oco1ord Jlaql uI slurod 1ua-rag.rp1E aJnlrEJ uear8 e lcalap .(eru sels 'puoces puv 'seJnlIEJa1s luanbasqns 01 Juelllsar aq lsnlu 1l '1sllg IocoloJduorl€urluJal e 3o u8rsap aq1 aleclldruoc sanssl o-^,rJ

'uorlcas snoraa"rd aql ur paqlJcsapsloaolo:d lrurruoc ao sarl:ado.rd aq1 uo spuadap locolorduorlEurruJal e Jo uollncaxa lcaJJoc aqJ 'alEls lualsls-uoc E ur uorlcEsu€J1 eql sal€urrrlJel ays leuorle:adod:aaa 1eq1 aalue:en8 lsnru IocoloJd uorleurut:a1 y

IocoloJd uoREulruraJ. paz-rleJluacao Y '"

'a1rs lue le pelJoqe sI uorlcesuEJl eq1 eJoJeqsa1uls alqE11lur[uocuou ldncco salls, lpuoTlerado iIE (Z)

'a1rs due le pelllruruoc sr uollcesuuJl ar{l aroJaqsa1E1s aIqEllluluroc,(dncco sa1ls IPuol1€rado 11e (t)

:(lqreuUxs] aas) satl:ad-o.rd Eurr*o11oJ eql 1-rqlqxe slocolo.rd Eut>1co1quou 11y

'alq€llrurtuocuou eJE sa1e1s Euruletual aql lsalelsalqEllrruruoc aJE a1E1s llruuloc aql pue a1e1s aredard

'punor aEessalx 1Buor1-rppe ue arrnba: Feru arnpe; a1ls qc€a 'asec lsJol{ aqlur 'l1a1eun1ro;u1-1 'uorlncaxa sy 8ur:np rncco saJnlr€Je11S OU Uaq/v! spunor aEessaur o,u1 .(11CeXa sa-rrnbar

luasa:d,!\ou er\r 1eq1 1oco1o;d aq; 'spunoJ or'r1 lseal1e sa:rnba.r 1oco1o:d lualllsar e 1eq1 panE;B a^€q al{

z'uolsrcap trlluruoJ lualsrsuocul uE aI"ruplnorlr 1r puB !z eJls lq paaraca-r esoql qlr,^a JualslsuoculaJa-^,r. palraca: 1r saEessarr: aql 1eql lcadsns 1ou plno'{!

t allsi 'uaqJ '(uorldrunsse alqpuosEar e) Iocol0rd aQl

3o SuruurEaq aq1 1e dn se,,. J alIS lEqJ aJe.ue lou sEl'r talIS 1eq1 atunssE sn la'i '(Z ayg u:o:;) eEBssaru alqolJl"tt'L

-ruo"uru a18uts e sallacal e alIS 'rtloN 's1re1 dlldruo'tduaql puB'sllturuoc'I e11S tuoJJ aFBssaru

"lqn4ll'1u- oc aqJ salraceJ 'saBessatu aTqnlJLluluocuou Jno spuas

e alls 'saqsprc uaql pu€ e alIS 01 a8essaru a7qnfl?lu-l(uac e Spuas I alrs areqrl orJ€uacs aq1 :aprsuoc 'ure8y'srualqo:d a11qns {raa oJ p€el uuc anssl puocas aqJ

'u,!rou>I aq 1ou derusalrs IBuoIl€Jado raqlo aql Jo sa1111uep1 eql '1oco1ord uorl-eururJal e olur ltlua uodn 'lu1ncr1:ud u1 'says JaqJo aql1o (,,u,nop,, ro ,,dn,, a r) sn1e1s leuotle:ado luarJnc eqll!ou{ lou saop alrs uearB e ualJo 1eq1 sI anssr puocas eqJ'uorlaesueJl aq1 Eutlltunuoc ralJe palleJ Z alIS aJaqirorJeuacs snoraald Jno ur esuc eqJ se.,t{ sIqJ '(spuno:aEessaur luanbasqns uI eledrcrl"red 01 alqEII€^Bunaq eJoJaraql pue) uorsrcep Jrluluoc e Eurleur -la11e l1a1e-rparulul peg deur alls i€uollurado ue 1uq1 sI anssl lsJUeqJ 'sanssr allqns ol,^1 dq palearldu:oc si 1oco1o:d uorl-eulluJal ,,punor-a1dr11nru,, lualllsal e ;o u8lsap aq;

IocoloJd uorlEurrrrJal lualllsau Y

'seJnlIeJals -(re:yqre o1 luaJUSaJ sr locolord uolleulruJel ,,punoJa18urs,, ou 1€t{1 (d11eruro3 u^roqs eq uec pue) a^l1lnluld1:ru3 st 11 'pallJluoc aq plnor olJEuacs a^oq€ eql Jo

"inrut ror:rru aql sI luql olJeuacs 8ur4co1q e uaql 'sa8es-sat1 a1qpJJl;tu1tto, luas peq sa1ls IIE JaIJE dluo yruruocoJ sB,t{ alnJ arll 3i 'aldruexa .rog 'aln: liluruoc aq1 8ul-Eueqc dq lsnqor aJolu apelu eq Jouuec 1oco1o:d aq;

'uo11

-o€su€r1 aq1 lcoiq Jsnru t alIS uaql'1urod srql lE sll€J

Z alls JI 'alrs palIBJ eql Jo a1Bls eql 01 sE 2 alrg sar-rantl

lI lllun paacord d1a;es louuec g altg 'd1rea13 'Z alrS ulorJaEessau a?qnlJL'tttlttocuol, E pue I a1ls uroJJ sa8Bssaurou pelracal eABq plno^! I alIS (E alrg ruolS) aEBssatua'lqol41zt"uncuou euo pue (1 a1g uro:3) aBessaur aTgo

-JJxzuutoc auo paAIaceJ elpq plno,tr Z elrs 'punol aEBssaul

lsru aql Jo pue aql 1Y 'Z alls o1 a8essaru a'lqnll?1aluac 2

Eurpuas JaIJE IIEJ I a1ls lei pue '1oco1ord uorleullxral aqJolur d.r1ua uodn a1e1s elq€llrturuoc e uI alls dluo aq1 aq

I alls la'I 'salls aaJql Euraio.rui olr€uacs Eurarollo3 aq1 urpal€Jlsuourap sI sB lsnqoJ d:aa 1ou sr locolord srq;

'JcarJoc sr loaolord uolleullrllal aldrurs aql 1Eq1 epn1c

-uoc ar!\ 'aJoJaJaI{J 'UOl1cESUeJl aql palJoqE sEq alls ou

lBr{1 satTdrul 11 'aJoluJaI{lJnJ :uoIJcESuEJl aI{1 1l1uIuoc uBcsells IIp 1eq1 sarldurr alrs lue 1E elels elqellllutuoc E JolcuBdncco leql aou{ a^r't uollcas ur ualrB slocolord yur-ruoc Eurlcolquou Jo sa11:adord aqJ uoJd 'a1E1s alqellllu-rrroc E u1 {11e11ur sr sluedrcrl:ud aq1 3o euo 3r dluo pue 1r

pallrurruoc sI uo1lc€sueJ1 aql 1Eql alJasqo all 1oco1o:daql Jo ssaulcaJJoc aql anE:e o1 p.le.r'r:o31qEteJls sI 1l

'a1€}S lr€rt{ aq1 Jo alpJs IEIlIUI

6Z

The protocol presented is an extension of the simpleprotocol. The same three messages - oborf, cornm'itt'able, and noncomm-tttable - will be used again in theflrst round and in aII subsequent rounds.

The sending of messages durlng the flrst roundproceeds as before: a site examines its transaction stateand sends the appropriate message. However, theactions triggered by the receipt of the messages differfrom before.

To deflne the remainder of the protocol we mustspe c ify:(1) the rules for the messages sent during the subse-

quent rounds,(2) the rules for moving to a flnal transaction state (i.e.

either commit or abort), and(3) the rules for terminating the protocol (this is

closely linked to (2)).

These rules are obviously inlerrelated, but we will treatthem sequentially.

The rules for sending messages are simpler and wiIIbe discussed flrst. The messages sent by a site in thesecond round and subsequent rounds will be determinedsolety by the messages received during the previousround. The reader is reminded that during a round asite sends the same message to all (operational) partici-pants, including itself. This message to itself, as anyother message, wiII be used in determining the nextround of messages.s

There are three cases which are treated in the nextthree paragraphs. The rules for sending messages aresummarized in Figure 2a.

The receipt of an abort message by a site during anyround implies that the sender has aborted the transac-tion. Therefore, in subsequent rounds the site will sendoborf messages,

The receipt of a single com,m-Lttable message duringthe flrst round implies that the transaction was committ-able at the sender, and therefore, it is committable at allsites. The receiver of lhe cornm.i.ttable message, beinginformed that the transaction is committable, shouldsend conrm-Lttable messages during ail subsequentrounds. Similarly, a com.rni.t.table message received dur-ing a subsequent round implies that all sites can commit,and will trigger the sending of comni,ttabla messages inalL of the later rounds.

If only noncontn-i.ttable messages are received dur-ing a round, then the site must send noacornrnittablemessages in the next round.

From the above three rules, we infer:temma 1. 0nce a site begins sending a cornmitt-able (abort) inessage, it wiII send that message inall subsequent rounds..We

now turn our attention to rules for committingand aborting the transaction. Clearly, if a site everreceives an abort message, it should immedialely abortthe transaction because the transaction has beenaborted at other sites (in particular, it was aborted bythe sender of the message). However, commit,ting atransaction is not so straightforward.

Recall that a major flaw with the simple terminationprotoeol is that a site commits after receiving a singlecornrniltable message. We require a rule analogous toproperty (1) of nonblocking commit protocols, which

sThis is the oaly ray that the previou state of the site plays a roleil determining the next state.

First message round:

type of transaction stateflnal abort statecommit,table stateall other states

Second and subsequent rounds:

messages from previous roundone or more abort messagesone or more committable messagesall noncommittable messages

message sentabortconrwdttablenonconvrdttable

message sentabortcornm,i,ttabLenoncomwvittable

a. Summary of rules for sending messages.

The transaction is terminated transaction if:

messages receiveda single abort messageall committable messages2 suecessive rounds of messageswhere all messages are rloncommittable and no site fails

final stateabortcommitabort

b. Summary of commit and termination rules.

Figure 2. Summary of the resilient decentralized termi-nation protocol.

states that all sites must be in a committable statebefore any site commits. This leads us to the followingrule:

Commit RuIe. A transaction is committed at asite only after the receipt of a round consistingentirely of commil,table me ssages.

Before continuing with the termination rules for theprotoeol, it will be instruetive to look at a "worst case"execution of the protocol. The execution is worst case inthe sense that the maximum number of message roundsis required before the transaction is committed. OnIythe rules previously discussed are used.

The worst case exeeution for flve participants isillustrated in Figure 3. (In the figure the messagesreceived by a site during a round comprise a vector,where the ith component is the message received fromthe ith site. C l, and y'f are abbreviations lor contm-itt-able, abort, and noncomrniltable. A dash (-) indicatesthat no message was received from that site.)

IniLially, the flrst site is the only one in a committ-able state. It fails after sending a single message that isaddressed to the second site. In general, during the keround the kth site fails after sending a single cammitt-able message (to the kth+1 site). Therefore, during eachround one more site becomes aware that the transactionis committable. This continues until the fifth round,where Site 5 is the sole remaining operational site and itcommits the transaction.

Ij

I

Ihm,.

30

i1

I

II

:sllnsar Bupr,ro11o3 aq1 arrnba.re.{,t'paqcEaJ sI a1E1s lue?slsuoc E 1€I{1 rirroqs oJ

'a.lnJ uorlEulluJa? aql Jo esnEcaq palJoqEaq IIIrt uolJaEsuEJl aq1 'asec Ja11EI aql uI '(spuno:u 1so[r 1e sarrnba.r srql puu) oJaz seqcEaJ Jaqrunueq1 aurl eq? dq PalEurluJal eq Il-r.r,a uorlcesuErl aq1'asEa JaluJoJ aq1 uI 'Jaqunu alrles aq1 q1]il Jncco IIIrraspunoJ ott\1 Jo 'punoJ qcEe I{1I,!I oJaz pJPrllol sasEaJcop.(llucruolouoru Joqlra seEessaLu alqDJl?lulu,ocuou Eul-puas salrs Jo Jeqrunu aqtr leq1 noqs E puE e sEluurerl

,'UoIlcIP€J1UoC € sIsrql urpEv saEEssaru a7qoJ7l11t1uoc puas Ilrir sellsIIu T+J punoJ uI 'aJoJaJarlJ 'punoJ }Eq1 EuIJnp(t uto:g) aFussaru a7qoJJLlilluoc E palracal salrsIIe 'r punoJ ur IIuJ lou plp I alls aculs 'J puno.r 8ur-rnp aSessaur alqolw1r.naoc E spuas r alIS 'Z ASSC

'(:)tg = (r+.r)'u rureic eql s?crpBrluoc srqJ'(uorldunsse lq) r puno: Eurrnp aEessaru alqe-llruruoc E pallacar 1I esn€caq aEessau: a?qn#Lllt:tuoc E pues IIil! 1I 'I+J punoJ ul 'J punoJ Eur.rnpa8Bssaru ayqnlJ?uaurocuou e spuas r alis 'I asBC

:J punoJ Eur.rnPr lq luas aEBssaur aql uo Eutpuadap sasecqnsol^1 aJe aJaq.L 'readde se8essaur alqnJl?1u1aoceJarlu asec aqJ ssncslp dluo 11rar all 'sa8essauralqolnluutocuou ueql raqlo sa8essaru ureluocr punor lel puu (r)rp = (r+.r)tX Jertr 'uollalperl-uoc lq spaacord luarun8re aqJ 'leuolleJado sr talrs 1Bq1 elunsse dyle:auaE Jo ssol lnoqlrrltr 'Joo-rd

'saEessatu alqoJJaluutoJuoz aJo,tt I +Jpue J spunoJ qloq Eurrnp r a1ls dq parrtaaa.r saEes

-saru IIB uer{l'O * (r)'U = (1+r)rp JI 'E EEua'I

r'J punoJ ur aEBs-seut a'lqDJJLlt-Lutocuau E luas e^Eq lsnIu 1l '1+JpunoJ ur aEessaru alqnlqlluluocuou e puas 01 ellsu roJ :I purrua.I uro:; .(11ca:tp saoIIoJ s1qJ 'JooJd

(r)ttt > (r +.r)'N 'z Euurarl:aAEq ai^

'J punoJ Bul.rnp r alrs o? sa8essaru alqDlf.nuuncuouSulpuas salls Jo les aq1 aq (:)!N 1a1 '1oco1o:d aq1 3o Eulu-u18aq aq1 1e s1uedrc11-red go raqrunu aql aq u larl

'ISJU Uo-rleunuJal .&Oqs ll-r.tid ar{, 'alEls lualsrsuoC e ursal€ulrurel 1I leql (a) pue 'salEuuural sr(e.a,r.1e 1oco1o.rd aq1lEq1 (I) lwoqs lsnur efi ssaulcaJJoc aleJlsuoruep oJ

1uaun8.ry ssarrlcaJroC

'q2 a.rnErg u1pazrJEruurns eJe selnJ uollEulluJal puE lfluu]oc eqJ

'puno: a Eessaru ieuorllppEaql uI aledrcrlred uorlcesueJl aq1 Eu11:oqe salrs aq1raqlaq.M. go aallcadsaul salrs IIE 1E uollcusueJl aql lroqe.(11en1uaaa 11:u. 1oco1o.rd aql :luaruacuequa,,acueruJoJ-:ad,, e dluo sr slql lEql aloN 'uo-{}cesu€Jl eq} Eurl:oqeJaIJE punor a8essaur leuoll-rppE auo ur aledrcrl.red o1 alrse a.rrnba,r s.{e,a,r.p li-rr\,r a$ 'aJoJaJar{J 'spunoJ ra11e1 u1 aled-rc-r1.red plnoqs puno: aEessau lsrg aq1 Euirnp uorlces-uErl aql Eurl.roqe salrs asoql 'salrs IIE le uorlcusueJl aq1

Jo uorlJoqe aq1 alpadxa oJ 'spunoJ a8essaur luanbasqnsq1r^r enulluoc IIIlt,r sJeqlo aql al-Iq,ll 'uollcEsuerl aqT uoqE.(lalerparuu:r 11.a,l, aEesseru ,JoQo uE Euyrlacar sluedrcrl:edasoql uaql 'punoJ lsJg aql ur saEpssaur Eutpuas alrqlrsIrEJ e1E1s lJoqE aq1 ur alrs eql JI 'a1e1s elqElllruluocuouE ur aq sluedrcrl:ed Euiuleruar ar{l lel pu€ 'a?els lroqe uEur alrs auo i(1uo aq {11er1rur aJaql 1aI 'elduexa Jod 'aur1etuEs ar{l lE uoqE 01 apraap 1ou r(eru se?rs IIE 1€r{1 aclloN'1oco1o:d eql 01 luatuecueque IBUU euo a{Eru uec elf,

'relel alnJ srql ,{3r1snI ll],u arg

'uo1?cEsuEJl el{l IJOqB ,(1a3es uec 1jr uaql'spunor aql uaaAlaq soJnlrpJ elrs ou slcalep 1lpue saEessaru alqoll.Llt urocuou lo spunoJ azrlssac-ans oi!1 sa^recar JeAe alls B JI 'alntr uoIlEuIuJaJ

'suorlcesueJl qcns elEulluJal 01a1n.r Eur.u.o11oJ aq1 asn III,ra erq. 'uorlcesuerl aql lroqB prreuorlEnlrs siql lcalap 01 alq€ aq Jsntu locolord aq; 'saEes-set;.: alqpJyllitluocuau Eurpuas aJe salls IIE aJaqrlr elels €o1 ssa:Eo.rd 01 uollcesue.rl eql JoJ alqrssod sr ?r 'JeAa,noH'u:a1qo:d ou sr eJarll ueql 'sells IIe urJoJ saEusseur a2go-J7?1ltutac selraceJ dilenluaaa .ro aEessatu ,"roqo auo lsEaIle sellacar ,{ilenluaaa alrs E JI 'locoloJd aq1 Surleutru-ra1 ,{11aa::oc ;o rualqord aql raplsuoc sn JaI rraoN

'1oco1ord uolleulruJal ?ualllsaJ aql Jo uollncaxa asBc lsJol[ '8 a.rnElg

aFessaur alEurs e Eutpuas ralJe sIIEJ alts (1) :916tr1!I

ii3---- OSIIVJ

N3--- (r)

vN--- NN3--,{/V11-- N,{N--11111,1d- /1r1t1N-

uou uou

s t{ils } f,JIs

OI]IVd CS'IIYJ

cs'itvJ(r)

NNNs- (I)

Mlt/ltrrV - lfilNlL

uou

8 T.LIS

CS'lIVJ

os'uv.d

uou

z fl.LIS

osTiY.{

Cf,']IVd

CE']IV.{

CSIIVd(r)

g punor

? punoJ

I punoJ

z punoJ

1 punoJ

elElsalqElFutuoc IEIl-Iul

I [J.IS

ou If,3f,usgcYssf,il

Lemma 4. During any message round, aborf andcorrtrni,ttable messages may not both be sent.

Proof. The proof for the flrst round followsdirectly from the properties of nonblocking com-mit protocols: lt is never the case that one site isin an abort state while another site is in a com-mitLable state.From the rules for sending messages, we knowthat a round can inelude a eertain type of mes-sage only if that message type was present in theprevious round. (This follows from the observa-tion that a given message type must be receivedby a site, before it wiII be sent by a site in thenext round.) By induction, a message type canappear in a Iater round only if it was present inthe flrst round. This observation proves the lem.-ma, r

Lemma 4 proves that it is never the case that somesites are lrying to abort the transaction by sending oborfmessages, while others are trying to commit the transac-tion by sending cornrnil,table messages. The cornmit ruleinsures that sites begin to commit only after ail opera-tion sites are ayrare that Lhe transacti.on is "committ-able." Finally, the properties of a nonblocking commitprotocol insure that no site has aborted the transactionafter a single site has entered a cornmittable state. Col-Iectively, these results imply the correctness of the pro-tocol.

5. Conclusionslfe have presented a termination protoeol that is

resilient to arbitrary site failures that do not partitionLhe network. In ISKEE81c] this protocol is extended tohandle network partitions.

The proposed termination protocol is an example ofa decentralized protocol. These protocols have severaladvantages over centralized protocols - notably theytend to be much simpler and easier to implement. Bothof these advantages are derived from the symmetryinherent in all decentralized protocols.

The major disadvantage of decentralized protocolsis the number of messages exchanged during a round(the number of messages is quadratic in the number ofparticipants). In network envi.ronments where eithercontrol messages are cheap or a broadcast facility isavailable or both (e.g. an ETHERNET), the message costis reasonable. Moreover, in realistic environments a sitefailure should be a rare event; therefore, the cost of theLermination protoeol should not be a significant issue.

Since tnessage rounds are costly, an importantdesign goal for any decentralized protocol is to minimizethe number of rounds. It is easy to show that any resi-Iient protocol requires a minimum of two messagerounds before it can commit a transaction and, in Lheworst case, requires an additional message round foreaeh failure detected ([SKEEBIc]). The proposed proto-col meets these lower bounds. In particular, it requiresexactly Lwo rounds when no additional site failures occurdurlng its execution. Furthermore, a worst case execu-tion of the protoco[ is extremely rare in practice.

Finally, the proposed protocol is an opfirrrrstic pro-tocol - it will commit the transaction whenever it is safeLo do so - and it can be used in conjunction with anynonblocking commit protocol. ]n environmenLs wheremessages are expensive, it is reasonable to run a cen-tralized commit proLocol and the proposed decentralizedtermination protocol.

REIERENCES

Alsberg, P. and Day, J., ''A Principle for ResiIient Sharing of Dislributed Resourees,',Proc. ?rld International Conference onSoftttare Engineering, San Francisco, Ca.,0ctober 1976.

EIIis, C.A., "A Robust Algorithm for UpdatingDuplicate Databases," Proceedings of theSecond BerkeLey lYorkshop on DistributedData Management and Computer Net,works,197?, pp. 146-158.

Gray, J. N., ''Notes on Database 0per.atingSystems," in @erating Sysf erns.. AnAduonc ed Course, Springer-VerIag, 19?9.

Hammer, M. and Shipman, D., "ReliabilityMechanisms for SDD-1: A System for Distri-buted Databases," Computer Corporation ofAmerica, Cambridge, Mass., July 1979.

Lampson, B. and Sturgis, H., "Crash Recoveryin a Distributed Storage System," Tech.ReporL, Computer Science Laborai.ory, XeroxParc, Palo AIto, California, 1976.

Lindsay, B.G. et aL., "Notes on DistributedDatabases", IBM Research Report, no. RJ25?1(July 1e7e).

Skeen, D. and M. Stonebraker, "A I'ormalModei of Crash Recovery in a DistributedSystem", IEEE Transacti.ons on SoftuareEngineerlng, (to appe ar).

Skeen, D., "Nonblocking Commit Protocols",SIGMOD Inter"natinnal Conf on Mo:nagem.entof Data, Ann Arbor, Michigan, 1981.

Skeen, D., Orosh Recouery [n a Di^stributedDatabase Managem"ent System., Ph.D. Thesis,EECS Department, University of California,Berkeley (in preparation).

Stonebraker, M., "Concurrency Control andConsistency of Multiple Copies in DistributedINGRES," IEEE TYonsactions on SoftttareEngineering, May 1979.

IALSB?61

[ELL]?71

IGRAY?e]

[HAMM?e]

ILAMP?6]

[LrNDTe]

lSKEEBlal

IsKEEBlb]

ISKEE81c]

IsT0N7el

32