1

Where Concurrency meets Distributed Algorithms

(for me)

Joint with:Kai EngelhardtRon van der Meyden

Yoram Moses Technion EE

2Yoram Moses

Distributed behavior is complex

Rampant nondeterminism: scheduling, timing, reliabilityMultitude of concurrent and co-dependent events Decentralized information and Control

Hence…

3Yoram Moses

Hence…

Basic tasks are hard to solve Mutual exclusion, Paxos, mail, agreement, voting,

spanning trees… Careful modeling is crucial

For verification, lower bounds

Distributed algorithms literature is fragmented Every interesting task demands significant attention Many different models Problems solved one by one on a per-model basis Similar solutions often repeated in different models “Bottom up” but individual solutions may or may not

be compatible.

4Yoram Moses

Focus of this talk

Top-Down considerations Refinement Cuts vs. global states

Seq. Composition of distributed algorithms Notions of safe composition Sealing and message reordering

5Yoram Moses

Main Points

Need tools for abstracting and unifying distributed solutions Both global states and cuts can matter Notions of safe sequential composition are interesting

6Yoram Moses

Goal No. 1: A Program Refinement Calculus

Desired: A top-down process by which a program is repeatedly replaced by a “more detailed” one, that satisfies the same requirements.

P refines Q if every execution of P is an execution of Q

Refinement for sequential programs is well established Back, Morgan et al.

7Yoram Moses

set y to max A[1..n]

sort A[1..n]

quicksort[A,n]

y:= A[n] ;

Example

; y:= A[n]

8Yoram Moses

Programs and Specifications

“Programs” can contain both specifications and concrete

commands Specifications are described using a pair [,] of a pre-

and post-condition, meaning: If started in a state satisfying the program will

terminate, and its final state will satisfy

Reactive behavior can be obtained by temporal statements in spec, and state abstraction by e.g., epistemic statements.

9Yoram Moses

Refinement Rules

Splitting Rule:[,] ; [ ,] refines [,]

Action Rules: x := 1 refines [true,x=1]

Refinement allows “correctness by design,” maintainability, and design for different models

10Yoram Moses

Review: Sequential ProgramsExecute between an initial and a final state

Define an (initial/final) relation on states

P

11Yoram Moses

A Distributed Program

Should execute between distributed states

A specification typically describes assumptions on the initial global state.

P1

P2

12Yoram Moses

But…

P1

P2

Q2

Q1

Distributed programs terminate along a cut

Even if P starts at a global state, Q will not.

13Yoram Moses

What is a distributed state?

A sequential state serves two roles: Transformation: Actions modify the

state Control: Programs execute between

statesClaim: In the distributed case Transformation: Actions modify global state Control: Programs execute between cuts

(c1,c2,…,cn) where ci ≤

14Yoram Moses

Cuts and Specs

Sequential composition amounts to “layering” P*Q:

The cuts on which layers compose are not necessarily consistentThe formulas in a spec [,] become cut formulae.

E.g., “x=0 y=2” , “consistent cut” …What is a good language to express these? (semantics for temporal, epistemic, …)

P Q

15Yoram Moses

Goal No. 2

Account for both aspects of a distributed state?

Allow for inconsistent cuts

Allow for multiplexing or forking?Havelund et al. 94

16Yoram Moses

Sequential Composition

Distributed applications often have a sequential structure:

1. Compute MST2. Elect a leader3. Perform a vote using this leader4. Act on the outcome of the vote…

These activities may overlap and interfere.Will individual solutions compose correctly?

17Yoram Moses

Communication-closed layers

Elrad & Francez 1982, Stomp & de Roever, Janssen Zwiers & Poel, …

Suppose that P= Q*L*R (so each Pi = Qi;Li;Ri ).L is communication-closed (CC) in P if, in all

executions of P, all communication actions in L are internal.

Communication closure in P may depend on all of P

Q L R

18Yoram Moses

Tail Communication Closure

Program L is TCC if L is communication closed in all programs P = L*Q Gerth and Shrira 86

If L and R are TCC then L*R is TCC. allows incremental CC design

Work on CC and TCC focused on reliable FIFO models. What happens beyond this?

19Yoram Moses

Example: REL – Reliable non-FIFO channels

Example: 2-process message transmission

In REL&FIFO the program MTij is TCC

Engelhardt & M DISC 2005

20Yoram Moses

Safe composition in REL

MTij is sealed by MTji : It will not interact with any later layer R.

21Yoram Moses

More Formally We define a programming language with a “phase” operator

(Q) means Q does not communicate with other layers

A notion of “P executes over” interval r[c,d] A refinement relation ≤ among programs Can define CC and Sealing:

L is a CC in P= Q*L*R: (Q*L*R) ≤ Q*(L)*R

L is TCC : (L*R) ≤ (L)*R for all R

Q seals L: (L*Q*R) ≤ (L)*Q*R for all R

22Yoram Moses

Sealing

Q seals L if L is a CC in L*Q*R for all R

Sealing allows incremental CC design

Sealing in REL is subtle: No universal seals (barriers) existOnly programs with no reachable communication commands are sealed by the empty programSealing related to Lamport causality

23Yoram Moses

Transitive Sealing

24Yoram Moses

An Unsealable Program

Any future communication can interfere

25Yoram Moses

Getting by with a little help…

Two-process variant of P’ was unsealable

26Yoram Moses

Sealing and Causality

Thm: Q seals L iff every rcv on a channel ij

in L causally precedes any snd on ij in or after Q (and L can consume all messages it sends)

Prop: A straight-line program has a succinct signaturethat can be used to decide issues of sealing

Thm: If L is sealable then it has a seal that uses O(n) messages (replacing O(n2) acknowledgements)

27Yoram Moses

Beyond Sealing: Fitting After

Program Q fits after L if layer L never communicates with Q in L*Q:

(L*Q) ≤ (L)*Q

Sealing implies fitting after, but the converse is false. Fitting after allows for incremental CC design. Fitting after is natural in FIFO models (channels are lossy and/or allow duplication) REL is the only imperfect model that does not require headers for composition Fekete and Lynch CONCUR 90, Engelhardt & M 05

28Yoram Moses

Conclusions

Models of concurrency are crucial for proving properties of programs and lower bounds or impossibility results There is a role for concurrency modeling in top-down design, refinement and the study of compositionality Sequential composition is of interest and has a rich structure