Correctness of Gossip-Based Membership under Message Loss

Maxim Gurevich, Idit Keidar

Technion

The Setting•Many nodes – n▫10,000s, 100,000s, 1,000,000s, …

•Come and go▫Churn

•Fully connected network▫Like the Internet

•Every joining node knows some others▫(Initial) Connectivity

Membership: Each Node Needs To Know Some Live Nodes•Applications▫Gossip partners ▫Unstructured overlay networks▫Gathering statistics

•Work best with random node samples▫Gossip algorithms converge fast▫Overlay networks are robust, good expanders▫Statistics are accurate

Membership Protocols•Each node has a view ▫Set of node ids▫Supplied to the application▫Used by membership protocol for maintenance▫Modeled as a directed graph

u v

w

v y w …

y

Desirable Properties•Randomness…•Holy grail for samples: IID▫Each sample uniformly distributed▫Each sample independent of other samples

Avoid spatial dependencies among view entries Avoid correlations between nodes

▫Good load balance among nodes

What About Churn? Desirable Properties Cont’d•Views should constantly evolve▫Remove failed nodes, add joining ones

•Views should evolve to IID from any state•Minimize temporal dependencies▫Dependence on the past should decay quickly ▫Useful for application requiring fresh samples

Do Existing Protocols Measure Up?

… … w …

Existing Work: Practical Protocols

•Studied only empirically▫Good load balance [Lpbcast, Jelasity et al 07] ▫Fast decay of temporal dependencies [Jelasity et al 07] ▫Induces spatial dependence

… … z …u v

w

v … w …

w

zExample: Push protocol

v … z …

Existing Work: Analysis

•Analyzed theoretically [Allavena et al 05, Mahlmann et al 06]

▫ Uniformity, load balance, spatial independence ▫ Unrealistic assumptions

Atomic actions with bi-directional communication No message loss

▫ No bounds on decay of temporal dependencies

… … z …… … w …u v

w

v … w …

w

zShuffle protocol

z

Our Contribution: Bridge This Gap •Formally specify desirable properties outlined

above•A practical protocol▫Tolerates message loss, churn, failures▫No complex bookkeeping for atomic actions

•Formally prove the desirable properties▫Including under message loss

… …

Send & Forget Membership•The best of push and shuffle•Some view entries may be empty

u v

w

v … w … u w

u w

S&F: Message Loss•Message loss▫Or no empty entries in v’s view

u v

w

u v

w

S&F: Compensating for Loss•Edges (view entries) disappear due to loss•Need to prevent views from emptying out•Keep the sent ids when too little ids in view

u v

w

u v

w

S&F: Advantages over Other Protocols

•No bi-directional communication▫No complex bookkeeping▫Tolerates message loss

•Simple▫Amenable to formal analysis

Easy to implement

•Proving all desirable properties▫Analytical: degrees distribution w/out loss

Used in setting duplication threshold▫Markov 1: degree distribution with loss▫Markov 2: Markov Chain of reachable global states

IID samples, Temporal Independence

•Hold even under (reasonable) message loss!

Key Contribution: Analysis

Analytic Degree Distribution

•Similar (better) to that of a random graph•Validated by a more accurate Markov model

0

0.05

0.1

0.15

0.2

0 10 20 30 40Node indegree

BinomialS&F AnalyticalS&F Markov

•Proving all desirable properties▫Analytical: degrees distribution w/out loss

Used in setting duplication threshold▫Markov 1: degree distribution with loss▫Markov 2: Markov Chain of reachable global states

IID samples, Temporal Independence

•Hold even under (reasonable) message loss!

Key Contribution: Analysis

…

Node Degree Markov Chain

•Numerically compute the stationary distribution

Transitions without loss

Transitions due to loss

State corresponding to isolated node

outdegree0 2 4 6

inde

gree

0

1

2

3

…

…

…

…

…

…

…

Results•Outdegree is bounded by

the protocol•Decreases with increasing

loss

• Indegree is not bounded• Low variance even under

loss•Typical overload at most 2x

0

0.05

0.1

0.15

0.2

0.25

0 20 40 60 80Node outdegree

loss=0loss=0.01loss=0.05loss=0.1

0

0.05

0.1

0.15

0.2

0.25

0 10 20 30 40Node indegree

loss=0loss=0.01loss=0.05loss=0.1

•Proving all desirable properties▫Analytical: degrees distribution w/out loss

Used in setting duplication threshold▫Markov 1: degree distribution with loss▫Markov 2: Markov Chain of reachable global states

IID samples, Temporal Independence

•Hold even under (reasonable) message loss!

Key Contribution: Analysis

Decay of Spatial Dependencies

• For uniform loss < 15%, dependencies decay faster than they are created

• 1 – 2loss rate fraction of view entries are independent▫ E.g., for loss rate of 3% more than 90% of entries are independent

u v

w

uv

w

u does not delete the sent ids

…

Temporal Independence•Dependence on past views decays within

O(log n view size) time•Use “expected conductance”•Ids travel fast enough▫Reach random nodes in O(log n) hops▫Due to “sufficiently many” independent ids in views -

previous slide

Conclusions•Formalized the desired properties of a

membership protocol•Send & Forget protocol▫Simple for both implementation and analysis

•Analysis under message loss▫Load balance▫Uniformity▫Spatial Independence▫Temporal Independence

Thank You