Discus: Lateral Error Correction for Time-Critical Multicast

Submission #162, 14 pages

ABSTRACTDiscus is a low-latency reliable multicast protocol designedfor time-critical clustered applications. It uses IP Multicast totransmit data and recovers from packet loss in end-hosts usingLateral Error Correction (LEC), a novel repair mechanism in-volving the exchange of XOR repairs between receivers andacross groups. In datacenters and clusters, application needsfrequently dictate large numbers of fine-grained overlappingmulticast groups. Existing multicast reliability schemes scalepoorly in such settings, providing latency of packet recoverythat depends inversely on the data rate within a single group- the lower the data rate, the longer it takes to recover lostpackets. LEC is insensitive to the rate of data in any onegroup and allows each node to split its bandwidth betweenhundreds to thousands of fine-grained multicast groups with-out sacrificing timely packet recovery. As a result, Discusprovides developers with a scalable, reliable and fast mul-ticast primitive to layer under high-level abstractions suchas publish-subscribe, group communication and replicatedservice/object infrastructures. We evaluate Discus on a 64-node cluster, showing that it scales up to 1024 groups pernode while recovering more than 97% of lost packets at anaverage of 25 milliseconds, with 1% loss rate.

1. INTRODUCTIONClusters and datacenters play an increasingly important rolein the contemporary computing spectrum, providing back-end computing and storage for a wide range of applications.The modern datacenter is typically composed of hundredsto thousands of inexpensive commodity blade-servers, net-worked via fast, dedicated interconnects. The software stackrunning on a single blade-server is a brew of off-the-shelfsoftware: commercial operating systems, proprietary mid-dleware, managed run-time environments and virtual ma-

chines, all standardized to reduce complexity and mitigatemaintenance costs.

The last decade has seen the migration of time-critical appli-cations to commodity clusters. Application domains rangingfrom computational finance to air-traffic control and mili-tary communication have been driven by scalability and costconcerns to abandon traditional real-time environments forCOTS datacenters. In the process, they give up conservative- and arguably unnecessary - guarantees of real-time perfor-mance for the promise of massive scalability and multiplenines of timely availability, all at a fraction of the runningcost. Delivering on this promise within expanding and in-creasingly complex datacenters is a non-trivial task, and nu-merous competing paradigms have emerged to enable the de-velopment of clustered applications. These include publish-subscribe and data distribution layers that allow systems toscale to hundreds of nodes, group communication and repli-cation infrastructures that support fault-tolerance and highavailability, and distributed object architectures designed toprovide programmers with easy-to-use abstractions.

The emergence of clustered applications built within theseparadigms has shifted intra-cluster data patterns from uni-cast traffic to many-to-many communication, ormulticast.For example, commercial publish-subscribe (pub-sub) sys-tems deliver published data to subscribers using per-topicmulticast groups. Multicast is also used in service and objectarchitectures to update replicated entities and query themredundantly. It is a fundamental building block in higherlevel distributed protocols that provide ordering and consis-tency guarantees. Multicast as an abstraction is ubiquitousin clustered settings; hence, it is critical that the technologyimplementing it bereliable - masking communication faultsquickly and inexpensively - andscalable.

Many high-quality reliable multicast protocols exist in theliterature, each of which scales metrics like throughput orlatency across dimensions as varied as group size [3, 9],numbers of senders [2], node and network heterogeneity [5],or geographical and routing distance [10, 15]. However,these protocols were primarily designed to extend the reachof multicast to massive networks, and may be unstable, inef-

1

ficient and ineffective when retrofitted to clustered settings.They are not optimized for the failure modes of clusters and,most importantly, are unable to cope with the unique scala-bility demands of time-critical fault-tolerant applications.

We posit that the critical dimension of scalability for clus-tered applications is thenumber of groups in the system.All the paradigms of distributed system-building mentionedabove use multicast in ways that induce large numbers ofoverlapping groups. For example, a computational financecalculator that uses a topic-based pub-sub system to sub-scribe to a fraction of the equities on the stock market willend up belonging in many multicast groups. Multiple ap-plications within a datacenter - each subscribing to differentsets of equities - can result in arbitrary patterns of groupoverlap. Similarly, entity (object or service) replication atfine granularity can result in a single node hosting many dif-ferent replicated entities. Replication driven by high-levelobjectives such as locality, load-balancing or fault-tolerancecan lead to distinct and overlapping replica sets - and hence,multicast groups - for each entity.

In this paper, we propose Discus, a time-critical reliablemulticast protocol designed to perform well in the multicastpatterns induced by clustered applications. Discus uses IPMulticast [8] to transmit data and recovers lost packets us-ing Lateral Error Correction (LEC), a novel error correctionmechanism in which XOR repair packets are probabilisticallyexchanged between receivers and across multicast groups.The latency of loss recovery in LEC depends inversely onthe aggregate rate of data in the system, rather than the ratein any one group. It performs equally well in any arbitraryconfiguration and cardinality of group overlap, allowing Dis-cus to scale to massive numbers of groups while retaining thebest characteristics of state-of-the-art multicast technology:even distribution of responsibility among receivers, insensi-tivity to group size, stable proactive overhead and gracefuldegradation of performance in the face of increasing lossrates.

1.1 Contributions• We motivate the problem of time-critical multicast in

clusters, and argue that the critical dimension of scal-ability in such settings is the number of groups in thesystem.

• We show that existing reliable multicast protocols haverecovery latency characteristics that are inversely de-pendent on the data rate in a group, and do not performwell when each node is in many low-rate multicastgroups.

• We propose Lateral Error Correction, a new reliabil-ity mechanism that allows packet recovery latency tobe independent of per-group data rate by intelligentlycombining the repair traffic of multiple groups. Wedescribe the design and implementation of Discus, a

reliable multicast protocol that uses LEC to achievemassive scalability in the number of groups in the sys-tem.

• We extensively evaluate the Discus implementation ona 64-node cluster, showing that it performs well withdifferent loss rates, tolerates bursty loss patterns, andis relatively insensitive to grouping patterns and over-laps - providing recovery characteristics that degradegracefully with the number of groups in the system, aswell as other conventional dimensions of scalability.

2. SYSTEM MODELWe consider patterns of multicast usage where each node isin many different groups of small to medium size (10 to 50nodes). Following the IP Multicast model, a group is definedas a set of receivers for multicast data, and senders do nothave to belong to the group to send to it. We expect each nodeto receive data from a large set of distinct senders, across allthe groups it belongs to.

Where does Loss occur in a Datacenter? Datacenternetworks have flat routing structures with no more than twoor three hops on any end-to-end path. They are typicallyover-provisioned and of high quality, and packet loss in thenetwork is almost non-existent. In contrast, datacenter end-hosts are inexpensive and easily overloaded; even with high-capacity network interfaces, the commodity OS often dropspackets due to buffer overflows caused by temporary trafficspikes or high-priority threads occupying the CPU. Hence,our loss model is one of short bursts of packets dropped atthe end-host receivers at varying loss rates.

Figure 1 confirms this claim: we send traffic bursts of varyinglength from one blade-server (running Linux 2.6.12) to itsneighbor on the same rack and observe the patterns of inducedloss: loss burst length is plotted on the y-axis, and traffic burstlength on the x-axis. We send roughly a thousand packetsat each traffic burst length before incrementing the length by10. As we increase the length of traffic bursts, losses aremore frequent and bursty.

The example in Figure 1 is admittedly simplistic - each in-coming burst of traffic arrives at the receiver within a verysmall number of milliseconds. However, we think it ap-proximates loss patterns at end-hosts quite well - we don’texpect to see significantly more loss since these are veryquick traffic bursts and represent worst-case scenarios. Onthe other hand, we don’t expect losses to be significantly lesssince we are running the receiver node empty and drainingpackets continuously out of the kernel, with zero contentionfor the CPU or the network, whereas the settings of interestto us will involve time-critical, possibly CPU-intensive ap-plications running on top of the communication stack. Theirregular nature of Figure 1 underlines the unpredictable na-ture of COTS operating systems - we have little control over

2

0

10

20

30

40

50

60

70

80

0 50 100 150 200

Los

s B

urst

Len

gth

(Pac

kets

)

Traffic Burst Length (Packets)

Loss Bursts

Figure 1: Loss at a Datacenter end-host: As we

increase traffic burst size from 1 to 200 across the

x-axis, losses are more frequent and bursty.

thread scheduling, resulting in bursts of arbitrary length.

We expect multi-group settings to intrinsically exhibit burstyincoming traffic - each node in the system is receiving datafrom multiple senders in multiple groups and it is likelythat the inter-arrival time of data packets at a node will varywidely, even if the traffic rate at one sender or group is steady.In some cases, burstiness of traffic could also occur due totime-critical application behavior - for example, imagine anupdate in the value of a stock quote triggering off activity inseveral system components, which then multicast informa-tion to a replicated central component. If we assume thateach time-critical component processes the update withina few hundred microseconds, and that inter-node socket-to-socket latency is around fifty microseconds (an actual numberfrom our experimental cluster), the final central componentcould easily see a sub-millisecond burst of traffic. In thiscase, the componentized structure of the application resultedin bursty traffic; in other scenarios, the application domaincould be intrinsically prone to bursty input. For example, afinancial calculator tracking a set of hundred equities withcorrelated movements might expect to receive a burst of ahundred packets in multiple groups almost instantaneously.

3. THE DESIGN OF A TIME-CRITICAL MUL-TICAST PRIMITIVE

In recent years, multicast research has focused almost exclu-sively on application-level routing mechanisms, or overlaynetworks ([6] is one example), designed to operate in thewide-area without any existing router support. The need foroverlay multicast stems from the lack of IP Multicast cover-age in the modern internet, which in turn reflects concerns ofadministration complexity, scalability, and the risk of mul-ticast ’storms’ caused by misbehaving nodes. However, thehomogeneity and comparatively limited size of datacenternetworks pose few scalability and administration challenges

to IP Multicast, making it a viable and attractive option insuch settings. In this paper, we restrict ourselves to a moretraditional definition of ’reliable multicast’, as a reliabilitylayer over IP Multicast. Given that the selection of data-center hardware is typically influenced by commercial con-straints, we believe that any viable solution for this contextmust be able to run on any mix of existing commodity routersand operating systems software; hence, we focus exclusivelyon mechanisms that act at the application-level, ruling outschemes which require router modification, such as PGM[11].

3.1 The Timeliness of (Scalable) Reliable Mul-ticast Protocols

Reliable multicast protocols typically consist of three logicalphases:transmission of the packet,discovery of packet loss,andrecovery from it. Recovery is a fairly fast operation; oncea node knows it is missing a packet, recovering it involvesretrieving the packet from some other node. However, inmost existing scalable multicast protocols, the time taken todiscover packet loss dominates recovery latency heavily inthe kind of settings we are interested in. The key insightis thatthe discovery latency of reliable multicast protocols is

usually inversely dependent on data rate: for existing proto-cols, the rate of outgoing data at a single sender in a singlegroup. Existing schemes for reliability in multicast can beroughly divided into the following categories:

ACK/timeout: RMTP [15], RMTP-II [16]. In this ap-proach, receivers send back ACKs (acknowledgements) tothe sender of the multicast. This is the trivial extension ofunicast reliability to multicast, and is intrinsically unscalabledue to ACK implosion; for each sent message, the sender hasto process an ACK from every receiver in the group [15]. Onework-around is to use ACK aggregation, which allows suchsolutions to scale in the number of receivers but requires theconstruction of a tree for every sender to a group. Also, anyaggregative mechanism introduces latency, leading to largertime-outs at the sender and delaying loss discovery; hence,ACK trees are unsuitable in time-critical settings.

Gossip-Based: Bimodal Multicast [3], lpbcast [9]. Re-ceivers periodically gossip histories of received packets witheach other. Upon receiving a digest, a receiver comparesthe contents with its own packet history, sending any packetsthat are missing from the gossiped history and requestingtransmission of any packets missing from its own history.Gossip-based schemes offer scalability in the number of re-ceivers per group, and extreme resilience by diffusing theresponsibility of ensuring reliability for each packet overthe entire set of receivers. However, they are not designedfor time-critical settings: discovery latency is equal to thetime period between gossip exchanges (a significant num-ber of milliseconds - 100ms in Bimodal Multicast [3]), andrecovery involves a further one or two-phase interaction asthe affected node obtains the packet from its gossip contact.

3

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1 2 4 8 16 32 64 128

Mill

isec

onds

Groups per Node (Log Scale)

SRM Average Discovery Delay

Figure 2: SRM’s Discovery Latency vs. Groups per

Node, on a 64-node cluster, with groups of 10 nodes

each. Error bars are min and max over 10 runs.

Thus Bimodal Multicast is bimodal in several senses: de-livery probability (the origin of the protocol name) but alsodelivery latency, and a loss can result in delays of severalhundred milliseconds.

NAK/Sender-based Sequencing: SRM [10]. Sendersnumber outgoing multicasts, and receivers discover packetloss when a subsequent message arrives. Loss discovery la-tency is thus proportional to the inter-send time at any singlesender to a single group - a receiver can’t discover a loss in agroup until it receives the next packet from the same senderto that group - and consequently depends on the sender’s datatransmission rate to the group. To illustrate this point, wemeasured the performance of SRM as we increased the num-ber of groups each node belonged in, keeping the throughputin the system constant by reducing the data rate within eachgroup - as Figure 2 shows, discovery latency of lost packetsdegrades linearly as each node’s bandwidth is increasinglyfragmented and each group’s rate goes down, increasing thetime between two consecutive sends by a sender to the samegroup. Once discovery occurs in SRM, lost packet recoveryis initiated by the receiver, which uses IP multicast (witha suitable TTL value); the sender (or some other receiver),responds with a retransmission, also using IP multicast.

Sender-based FEC [13, 17]: Forward Error Correctionschemes involve multicasting redundant error correction in-formation along with data packets, so that receivers can re-cover lost packets without contacting the sender or any othernode. FEC mechanisms involve generatingc repair packetsfor every r data packets, such that anyr of the combinedset ofr + c data and repair packets is sufficient to recoverthe originalr data packets; we term this(r, c) parameter therate-of-fire. FEC mechanisms have the benefit oftunabil-

ity, providing a coherent relationship between overhead andtimeliness - the more the number of repair packets generated,

the higher the probability of recovering lost packets from theFEC data. Further, FEC based protocols are very stable un-der stress, since recovery does not induce large degrees ofextra traffic. As in NAK protocols, the timeliness of FECrecovery depends on the data transmission rate of a singlesender in a single group; the sender can send a repair packetto a group only after sending outr data packets to that group.Traditionally, sender-based FEC schemes tend to use com-plex encoding algorithms to obtain multiple repair packets,placing computational load on the sender. However, fasterencodings have been proposed recently, such as Tornadocodes [4], which make sender-based FEC a very attractiveoption in multicast applications involving a single, dedicatedsender; for example, software distribution or internet radio.

Receiver-based FEC [2]: Of the above schemes, ACK-based protocols are intrinsically unsuited for time-criticalmulti-sender settings, while sender-based sequencing andFEC limit discovery time to inter-send time at a single senderwithin a single group. Ideally, we would like discoverytime to be independent of inter-send time, and combine thescalability of a gossip-based scheme with the tunability ofFEC. Receiver-based FEC, first introduced in the Slingshotprotocol [2], provides such a combination: receivers generateFEC packets from incoming data and exchange these withother randomly chosen receivers. Since FEC packets aregenerated fromincoming data at a receiver, the timelinessof loss recovery depends on the rate of data multicast inthe

entire group, rather than the rate at any given sender, allowingscalability in the number of senders to the group.

Slingshot is aimed at single-group settings, recovering frompacket loss in time proportional to that group’s data rate. Ourgoal with Discus is to achieve recovery latency dependent onthe rate of data incoming at a nodeacross all groups. Es-sentially, we want recovery of packets to occur as quickly ina thousand 10 Kbps groups as in a single 10 Mbps group,allowing applications to divide node bandwidth among thou-sands of multicast groups while maintaining time-criticalpacket recovery. To achieve this, we introduce Lateral ErrorCorrection, a new form of receiver-generated FEC that prob-abilistically combines receiver-generated repair traffic acrossmultiple groups to drive down packet recovery latencies.

4. LATERAL ERROR CORRECTION ANDTHE DISCUS PROTOCOL

In Discus, each node belongs to a number of groups, andreceives data multicast within any of them. The basic opera-tion of the protocol involves generating XORs from incom-ing data and exchanging them with other randomly selectednodes. Discus operates using two different packet types:data packets - the actual data multicast within a group - andrepair packets, which contain recovery information for mul-tiple data packets. Figure 3 shows the structure of thesetwo packet types. Each data packet header contains a packetidentifier - a (sender, group, sequence number) tuple that

4

Application Payload

Packet ID:(sender, group, seqno)

XOR of Data Packets

List of Data Packet IDs:(sender1, group1, seqno1)(sender2, group2, seqno2)...

App

licat

ion

MTU

: 102

4 by

tes

Less

than

Net

wor

k M

TU

Data Packet :

Repair Packet :

Figure 3: Discus Packet Structure

identifies it uniquely. A repair packet contains an XOR ofmultiple data packets, along with a list of their identifiers -we say that the repair packet iscomposed from these datapackets, and that the data packets areincluded in the repairpacket. An XOR repair composed fromr data packets allowsrecovery of one of them, if all the otherr − 1 data packetsare available; the missing data packet is obtained by simplycomputing the XOR of the repair’s payload with the otherdata packets.

At the core of Discus is the LEC engine1 running at each nodethat decides on the composition and destinations of repairpackets, creating them from incoming data across multiplegroups. The operating principle behind LEC is the notionthat repair packets sent by a node to another node can becomposed from data in any of the multicast groups that arecommon to them. This allows recovery of lost packets atthe receiver of the repair packet to occur within time that’sinversely proportional to the aggregate rate of data in all thesegroups. Figure 4 illustrates this idea:n1 has groupsA andB

in common withn2, and hence it can generate and dispatchrepair packets that contain data from both these groups.n1

needs to wait only until it receives 5 data packets in eitherA

or B before it sends a repair packet, allowing faster recoveryof lost packets atn2.

While combining data from different groups in outgoing re-pair packets drives down recovery time, it tampers with thecoherent tunability that single group receiver-based FEC pro-vides. Therate-of-fire parameter in receiver-based FEC pro-vides a clear, coherent relationship between overhead andrecovery percentage; for everyr data packets,c repair pack-ets are generated in the system, resulting in some computableprobability of recovering from packet loss. The challenge for

1LEC refers to the algorithmic technique of combining anddisseminating repair traffic across multiple groups, and Dis-cus is the protocol that implements this algorithm in a clus-tered setting; we make this distinction since we believe LECto have applicability to other networking environments, suchas wireless mesh networks.

n1

n2

n3 n4

INC

OM

ING

DA

TA P

AC

KE

TS

Per-Group Repairs vs Multi-Group Repairs

Repair Packet I:(A1, A2, A3, B1, B2)

Repair Packet II:(A4, A5, B3, B4, A6)

n1 n2

A1

A2A3

B2

B1

A4

A5

B4

B3

A6B5

A1

A2A3

B2

B1

A4

A5

B4

B3

A6

Repair Packet I:(A1, A2, A3, A4, A5)

Repair Packet II:(B1,B2,B3,B4,B5)

B5

n1 n2

Figure 4: Lateral Error Correction in 2 groups

LEC is to combine repair traffic for multiple groups whileretaining per-group overhead and recovery percentages, sothat each individual group can maintain its own rate-of-fire.To do so, we abstract out the essential properties of receiver-based FEC that we wish to maintain:1. Coherent, Tunable Per-Group Overhead: For every datapacket that a node receives in a group with rate-of-fire(r, c),it sends outan average of c repair packets including that datapacket to other nodes in the group.2. Randomness: Destination nodes for repair packets arepickedrandomly, with no node receiving more or less repairsthan any other node, on average.

LEC supports overlapping groups with the samer compo-nent and differentc values in their rate-of-fire parameter. InLEC, the rate-of-fire parameter is translated into the follow-ing guarantee: For every data packetd that a node receives ina group with rate-of-fire(r, c), it selects an average ofc nodesfrom the group randomly and sends each of these nodes ex-actly one repair packet that includesd. In other words, thenode sends an average ofc repair packets containingd to thegroup. In the following section, we describe the algorithmthat LEC uses to compose and dispatch repair packets whilemaintaining this guarantee.

5

2

1

4

Figure 5: n1 belongs to groups A, B, C: it divides

them into disjoint regions abc, ab, ac, bc, a, b, c

4.1 Algorithm OverviewDiscus is a symmetric protocol - exactly the same LEC algo-rithm and supporting code runs at every node - and hence, wecan describe its operation from the vantage point of a singlenode,n1.

4.1.1 RegionsThe LEC engine running atn1 divides n1’s neighborhood- the set of nodes it shares one or more multicast groupswith - into regions, and uses this information to construct anddisseminate repair packets. Regions are simply the disjointintersections of all the groups thatn1 belongs to. Figure 5shows the regions in a hypothetical system, wheren1 is inthree groups,A, B andC. We denote groups by upper-caseletters and regions by the concatenation of the group namesin lowercase; i.e,abc is a region formed by the intersectionof A, B and C. In our example, the neighborhood set ofn1 is carved into seven regions:abc, ac, ab, bc, a, b andc, essentially the power set of the set of groups involved.Readers may be alarmed that this transformation results inan exponential number of regions, but this is not the case;we are only concerned with non-empty intersections, thecardinality of which is bounded by the number of nodes inthe system, as each node belongs to exactly one intersection(see Section 4.1.4). Note thatn1 does not belong to groupDand is oblivious to it; it observesn2 as belonging to regionb,rather thanbd, and is not aware ofn4’s existence.

4.1.2 Selecting targets from regions, not groupsInstead of selecting targets for repairs randomly from theentire group, LEC selects targets randomly fromeach region.The number of targets selected from a region is set such that:1. It is proportional to the size of the region2. The total number of targets selected, across regions, is

1

1

Aab

A abc

A a

Aac

Ax

A

A

A

Figure 6: n1 selects proportionally sized chunks of

cA from the regions of A

equal to thec value of the groupHence, for a given groupA with rate-of-fire(r, cA), the num-ber of targets selected by LEC in a particular region, sayabc,is equal tocA ∗ |abc|

|A| , where|x| is the number of nodes in theregion or groupx. We denote the number of targets selectedby LEC in regionabc for packets in groupA ascabc

A . Figure6 showsn1 selecting targets for repairs from the regions ofA.

Note that LEC may pick a different number of targets from aregion for packets in a different group; for example,cabc

A dif-fers fromcabc

B . Selecting targets in this manner also preservesrandomness of selection; if we rephrase the task of target se-lection as a sampling problem, where a random sample ofsize c has to be selected from the group, selecting targetsfrom regions corresponds tostratified sampling, an existingtechnique from statistical theory.

4.1.3 Why select targets from regions?Selecting targets from regions instead of groups allows LECto construct repair packets from multiple groups; since weknow that all nodes in regionab are interested in data fromgroupsA andB, we can create composite repair packets fromincoming data packets in both groups and send them to nodesin that region.

6

Single-group receiver-based FEC [2] is implemented using asimple construct called arepair bin, which collects incomingdata within the group. When a repair bin reaches a thresh-old size ofr, a repair packet is generated from its contentsand sent toc randomly selected nodes in the group, afterwhich the bin is cleared. Extending the repair bin constructto regions seems simple; a bin can be maintained for eachregion, collecting data packets received in any of the groupscomposing that region. When the bin fills up to sizer, itcan generate a repair packet containing data from all thesegroups, and send it to targets selected from within the region.

Using per-region repair bins raises an interesting question: ifwe construct a composite repair packet from data in groupsA, B, andC, how many targets should we select from regionabc for this repair packet -cabc

A , cabcB , or cabc

C ? One possiblesolution is to pick the maximum of these values. Ifcabc

A ≥cabc

B ≥ cabcC , then we would selectcabc

A . However, a datapacket in groupB, when added to the repair bin for the regionabc would be sent to an average ofcabc

A targets in the region;resulting in more repair packets containing that data packetsent to the region than required (cabc

B ), which results in morerepair packets sent to the entire group. Hence, more overheadis expended per data packet in groupB than required by its(r, cB) value; a similar argument holds for data packets ingroupC as well.

Instead, we choose theminimum of values; this, as expected,results in a lower level of overhead for groupsA andB thanrequired, resulting in a lower fraction of packets recoveredfrom LEC. To rectify this we send the additional compen-sating repair packets to the regionabc from the repair binsfor regionsa andb. The repair bin for regiona would se-lect cabc

A − cabcC ) destinations, on an average, for every repair

packet it generates; this is in addition to thecaA destinations

it selects from regiona.

A more sophisticated version of the above strategy involvesiteratively obtaining the required repair packets from regionsinvolving the remaining groups; for instance, we would havethe repair bin forab select the minimum ofcabc

A and cabcB

- which happens to becabcB - from abc, and then have the

repair bin fora select the remainder value,cabcA − cabc

B , fromabc. Algorithm 1 illustrates the final approach adopted byLEC, and Figure 7 shows the output of this algorithm foran example scenario. Note that a repair bin selects a non-integral number of nodes from an intersection by alternatingbetween its floor and ceiling probabilistically, in order tomaintain the average at that number.

4.1.4 ComplexityThe algorithm described above is run every time nodes joinor leave any of the multicast groups thatn1 is part of. Thealgorithm has complexityO(I · d), whereI is the numberof populated regions (i.e, with one or more nodes in them),andd is the maximum number of groups that form a region.

|c|=5

|a|=5 |b|=1|ab|=2

|bc|=7|ac|=3

|abc|=10

1

Figure 7: Mappings between repair bins and re-

gions: the repair bin for ab selects 0.4 targets from

region ab and 0.8 from abc for every repair packet.

Here, cA = 5, cB = 4, and cC = 3.

Algorithm 1 Algorithm for Setting Up Repair Bins1: Code at node ni:

2: upon Change in Group Membershipdo

3: while L not empty {L is the list of regions}do

4: Select and remove the regionRi = abc...z from L

with highest number of groups involved (break tiesin any order)

5: SetRt = Ri

6: while Rt 6= ε do

7: set cmin to min(cRtA , cRt

B ...), where{A,B,...} isthe set of groups formingRt

8: Set number of targets selected byRi’s repair binfrom regionRt to cmin

9: RemoveG fromRt, for all groupsG wherecRtG =

cmin

10: For each remaining groupG′ in Rt, set cRtG′ =

cRtG′ − cmin

7

Note thatI atn1 is bounded from above by the cardinality ofthe set of nodes that share a multicast group withn1, sinceregions are disjoint and each node exists in exactly one ofthem.d is bounded by the number of groups thatn1 belongsto.

4.2 Implementation DetailsOur implementation of Discus is in Java. Below, we discussthe details of the implementation, along with the performanceoptimizations involved - some obvious and others subtle.

4.2.1 Repair BinsA Discus repair bin is a lightweight structure holding an XORand a list of data packets, and supporting anadd operationthat takes in a data packet and includes it in the internal state.The repair bin is associated with a particular region, receivingall data packets incoming in any of the groups forming thatregion. It has a list of regions from which it selects targetsfor repair packets; each of these regions is associated with avalue, which is the average number of targets which must beselected from that region for an outgoing repair packet. Inmost cases, as shown in Figure 7, the value associated with aregion is not an integer; as mentioned before, the repair binalternates between the floor and the ceiling of the value tomaintain the average at the value itself. For example, in theexample in Figure 7, the repair bin forabc has to select1.2

targets fromabc, on average; hence, it generates a randomnumber between0 and 1 for each outgoing repair packet,selecting1 node if the random number is more than0.2, and2 nodes otherwise.

4.2.2 Staggering for Bursty LossesA crucial algorithmic optimization in Discus isstaggering -also known as interleaving in literature [17] - which providesresilience to bursty losses. Given a sequence of data pack-ets to encode, a stagger of2 would entail constructing onerepair packet from the 1st, 3rd, 5th... packets, and anotherrepair packet from the 2nd, 4th, 6th... packets. The stag-ger value defines the number of repairs simultaneously beingconstructed, as well as the distance in the sequence betweentwo data packets included in the same repair packet. Con-sequently, a stagger ofi allows us to tolerate a loss burst ofsizei while resulting in a proportional slowdown in recoverylatency, since we now have to wait forO(i ∗ r) data packetsbefore despatching repair packets.

In conventional sender-based FEC, staggering is not a veryattractive option, providing tolerance to very small bursts atthe cost of multiplying the already prohibitive loss discoverylatency. However, LEC recovers packets so quickly that wecan tolerate a slowdown of a factor of ten without leaving thetens of milliseconds range; additionally, a small stagger at thesender allows us to tolerate very large bursts of lost packetsat the receiver, especially since the burst is dissipated amongmultiple groups and senders. Discus implements a stagger ofi by the simple expedient of duplicating each logical repair

bin intoi instances; when a data packet is added to the logicalrepair bin, it is actually added to a particular instance of therepair bin, chosen in round-robin fashion. Instances of aduplicated repair bin behave exactly as single repair bins do,generating repair packets and sending them to regions whenthey get filled up.

4.2.3 Multi-Group ViewsEach Discus node has amulti-group view, which containsmembership information about other nodes in the system thatshare one or more multicast groups with it. In traditionalgroup communication literature, aview is simply a list ofmembers in a single group [18]; in contrast, a Discus node’smulti-group view divides the groups that it belongs to into anumber of regions, and contains a list of members lying ineach region. Discus uses the multi-group view at a node todetermine the sizes of regions and groups, to set up repairbins using the LEC algorithm. Also, the per-region listsin the multi-view are used to select destinations for repairpackets. The multi-group view atn1 - and consequently thegroup and intersection sizes - does not includen1 itself.

4.2.4 Membership and Failure DetectionDiscus can plug into any existing membership and failuredetection infrastructure - as long as it is provided with rea-sonably up-to-date views of per-group membership by someexternal service. In our implementation, we use simple ver-sions of Group Membership (GMS) and Failure Detection(FD) services, which execute on high-end server machines.If the GMS receives a notification from the FD that a nodehas failed, or it receives a join/leave to a group from a node,it sends an update to all nodes in the affected group(s). TheGMS is not aware of regions; it maintains conventional per-group lists of nodes, and sends per-group updates when mem-bership changes. For example, if noden55 joins groupA,the update sent by the GMS to every node inA would bea 3-tuple: (Join, A, n55). Individual Discus nodes processthese updates to construct multi-group views relative to theirown membership.

Since the GMS does not maintain region data, it has to scaleonly in the number of groups in the system; this can be easilydone by partitioning the service on group id and running eachpartition on a different server. For instance, one machine isresponsible for groupsA andB, another forC andD, andso on. Similarly, the FD can be partitioned on a topolog-ical criterion; one machine on each rack is responsible formonitoring other nodes on the rack by pinging them peri-odically. For fault-tolerance, each partition of the GMS canbe replicating on multiple machines using a strongly con-sistent protocol like Paxos. The FD can have a hierarchicalstructure to recover from failures; a smaller set of machinesping the per-rack failure detectors, and each other in a chain.While this semi-centralized solution would be inappropriatein a high-churn environment, such as a WAN or ad-hoc set-ting, we believe that it is reasonable in datacenters, where

8

membership is relatively stable. Crucially, the protocol itselfdoes not need consistent membership, and degrades grace-fully with the degree of inconsistency in the views; if a failednode is included in a view, performance will dip fractionallyin all the groups it belongs to as the repairs sent to it by othernodes are wasted.

4.2.5 PerformanceSince Discus creates LEC information from each incom-ing data packet, the critical communication path that a datapacket follows within the protocol is vital in determiningeventual recovery times and the maximum sustainable through-put. XORs are computed in each repair bin incrementally, aspackets are added to the bin. A crucial optimization used ispre-computation of the number of destinations that the repairbin sends out a repair to, across all the regions that it sendsrepairs to: Instead of constructing a repair and deciding onthe number of destinations once the bin fills up, the repairbin precomputes this number and constructs the repair onlyif the number is greater than 0. When the bin overflows andclears itself, the expected number of destinations for the nextrepair packet is generated. This restricts the average numberof two-input XORs per data packet toc (from the rate-of-fire) in the worst case - which occurs when no single repairbin selects more than 1 destination, and hence each outgoingrepair packet is a unique XOR.

4.2.6 Buffering and Loss ControlLEC - like any other form of FEC - works best when losses arenot in concentrated bursts. Discus maintains an application-level buffer within the VMS with the aim of minimizing in-kernel losses, serviced by a separate thread that continuouslydrains packets from the kernel. As we saw in figure 1, thismethod limits most loss bursts to less than ten packets, upto a threshold traffic burst size. If memory at end-hosts isconstrained and the application-level buffer is bounded, weuse customized packet-drop policies to handle overflows: arandomly selected packet from the buffer is dropped andthe new packet is accommodated instead. In practice, thisresults in a sequence of almost random losses from the buffer,which are easy to recover using FEC traffic. Whether theapplication-level buffer is bounded or not, it ensures thatpacket losses in the kernel are reduced to short bursts thatoccur only during periods of overload or CPU contention.We evaluate Discus against loss bursts of upto 100 packets,though in practice we expect the kind of loss pattern shownin 1, where few bursts are greater than 20-30 packets, evenwith highly concentrated traffic spikes.

4.2.7 NAK Layer for 100% RecoveryDiscus recovers a high percentage of lost packets via theproactive LEC traffic; for certain applications, this proba-bilistic guarantee of packet recovery is sufficient and evendesirable in cases where data ’expires’ and there is no utilityin recovering it after a certain number of milliseconds. How-ever, the majority of applications require 100% recovery of

lost data, and Discus uses a reactive NAK layer to providethis guarantee. If a receiver does not recover a packet throughLEC traffic within a timeout period after discovery of loss,it sends an explicit NAK to the sender and requests a re-transmission. While this NAK layer does result in extrareactive repair traffic, two factors separate it from traditionalNAK mechanisms: firstly, recovery can potentially occurvery quickly - within a few hundred milliseconds - since foralmost all lost packets discovery of loss takes place withinmilliseconds through LEC traffic. Secondly, the NAK layeris meant solely as a backup mechanism for LEC and respon-sible for recovering a very small percentage of total loss, andhence the extra overhead is minimal.

4.2.8 OptimizationsDiscus maintains a buffer of unusable repair packets thatenable it to utilize incoming repair packets better. If onerepair packet is missing exactly one more data packet thananother repair packet, and both are missing at least one datapacket, Discus obtains the extra data packet by XORing thetwo repair packets. Also, it maintains a list of unusable repairpackets which is checked intermittently to see if recent datapacket recoveries and receives have made any old repairpackets usable.

5. EVALUATIONWe evaluated our Java implementation of Discus on a 64-node cluster, comprising of four racks of 16 nodes each,interconnected via two levels of switches. Each node has asingle 1.3 Ghz CPU with 512 Mb RAM, runs Linux (kernelversion: 2.6.12) and has two 100 Mbps network interfaces,one which we use for control purposes and the other for ex-perimental traffic. Typical socket-to-socket latency withinthe cluster is around 50 microseconds. In the following ex-periments, for a given loss rateL, three different loss modelsare used:· uniform - also known as the Bernoulli model [19] - refersto dropping packets with uniform probability equal to theloss rateL.· bursty involves dropping packets in equal bursts of lengthb. The probability of starting a loss burst is set so that eachburst is of exactlyb packets and the loss rate is maintainedat L. This is not a realistic model but allows us to preciselymeasure performance relative to specific burst lengths.·markov drops packets using a simple 2-state markov chain,where each node alternates between a lossy and a losslessstate, and the probabilities are set so that the average lengthof a loss burst ism and the loss rate isL, as described in [19].

In experiments with multiple groups, nodes are assigned togroups at random, and the following formula is used to relatethe variables in the grouping pattern:n ∗ d = g ∗ s, wheren is the number of nodes in the system (set to 64 in all theexperiments),d is the degree of membership, i.e. the numberof groups each node joins,g is the number of groups in thesystem, ands is the average size of each group. For example,

9

0

10

20

30

40

50

0 50 100 150 200 250

Rec

over

y Pe

rcen

tage

Milliseconds

96.8% LEC + 3.2% NAK

0

10

20

30

40

50

0 50 100 150 200 250

Rec

over

y Pe

rcen

tage

Milliseconds

92% LEC + 8% NAK

0

10

20

30

40

50

0 50 100 150 200 250

Rec

over

y Pe

rcen

tage

Milliseconds

84% LEC + 16% NAK

(a) 10% Loss Rate (b) 15% Loss Rate (c) 20% Loss Rate

Figure 8: Distribution of Recoveries: LEC + NAK for varying degrees of loss

to test the protocol in a 16-node setting where each node joins512 groups and each group is of size 8, we would setg - thetotal number of groups in the system - to16∗512

8≈ 1024. Each

node is then assigned to 512 randomly picked groups out of1024. Hence, the grouping patterns for each experiment iscompletely represented by a(n, d, s) tuple.

For every run, we calibrate the sending rate at a node suchthat the total system throughput of incoming messages is64000 packets per second, or 1000 packets per node persecond. Data packets are 1K bytes in size. Each point inthe following graphs - other than Figure 8, which shows thedistributions for single runs - is an average of five runs. Arun lasts 30 seconds and produces≈ 2 million receive eventsin the system.

5.1 Distribution of Recoveries in DiscusFirst, we provide a snapshot of what typical packet recoverytimelines look like in Discus. Earlier, we made the asser-tion that Discus discovers the loss of almost all packets veryquickly through LEC traffic, recovers a majority of these in-stantly and recovers the remainder using an optional NAKlayer. In Figure 8, we show the histogram of packet recov-ery latencies for a 16-node run with degree of membershipd = 128 and group sizes = 10. We use a simplistic NAKlayer that starts unicasting NAKs to the original sender of themulticast 100 milliseconds after discovery of loss, and retriesat 50 millisecond intervals. Figure 8 shows three scenarios:with loss rates 10%, 15%, and 20%, different fractions ofpacket loss are recovered through LEC and the remaindervia reactive NAKs. These graphs illustrate the meaning ofthe LEC recovery percentage: if this number is high, morepackets are recovered very quickly without extra traffic inthe initial segment of the graphs, and less reactive overheadis induced by the NAK layer. Importantly, even with a re-covery percentage as low as 84% in Figure 8(c), we are ableto recover all packets within 250 milliseconds with a crudeNAK layer due to early LEC-based discovery of loss. Forthe remaining experiments, we will switch the NAK layer

0

20

40

60

80

100

1 2 3 4 5 6 7 8 10000

15000

20000

25000

30000

Perc

enta

ge

Rec

over

y L

aten

cy (m

icro

seco

nds)

c

Changing c, Fixed r=8

Recovery %Latency

Overhead %

0

20

40

60

80

100

2 3 4 5 6 7 8 9 10 10000

15000

20000

25000

30000

Perc

enta

ge

Rec

over

y L

aten

cy (m

icro

seco

nds)

r

Changing r, Fixed c=r/2

Recovery %Latency

Overhead %

Figure 9: Tuning LEC : tradeoff points available

between recovery %, timeliness and overhead by

changing the rate-of-fire (r, c). Recovery and Over-

head %s are plotted against the left y-axis, and Av-

erage Recovery Time on on the right y-axis.

10

90

92

94

96

98

100

2 4 8 16 32 64 128 256 512 1024

LE

C R

ecov

ery

Perc

enta

ge

Groups per Node (Log Scale)

Recovery %

0

10000

20000

30000

40000

50000

2 4 8 16 32 64 128 256 512 1024

Mic

rose

cond

s

Groups per Node (Log Scale)

Average Recovery Latency

Figure 10: Scalability in Groups

off and focus solely on LEC performance; also, since wefound this distribution of recovery latencies to be fairly rep-resentative, we present only the percentage of lost packetsrecovered using LEC and the average latency of these recov-eries. Experiment Setup:(n = 16, d = 128, s = 10), LossModel: Uniform,1%.

5.2 Tunability of LEC in multiple groupsThe Slingshot protocol [2] illustrated the tunability of receiver-generated FEC for a single group; we include a similar graphfor Discus in Figure 9, showing that the rate-of-fire parameter(r, c) provides a knob to tune LEC’s recovery characteristics.In Figure 9.a, we can see that increasing thec value for con-stantr = 8 increases the recovery percentage and lowersrecovery latency by expending more overhead - measured asthe percentage of repair packets to all packets. In Figure 9.b,we see the impact of increasingr, keeping the ratio ofc tor - and consequently, the overhead - constant. For the restof the experiments, we set the rate-of-fire at(r = 8, c = 5).Experiment Setup:(n = 64, d = 128, s = 10), Loss Model:Uniform, 1%.

0

50

100

150

200

250

300

350

400

450

2 4 8 16 32 64 128 256 512 1024

Mic

rose

cond

s

Groups per Node (Log Scale)

Updating Repair BinsData Packet Accounting

Waiting in BufferXOR

0

1

2

3

4

5

2 4 8 16 32 64 128 256 512 1024

Num

ber

Groups per Node (Log Scale)

XORs per Data Packet

Figure 11: CPU time and XORs per data packet

5.3 ScalabilityNext, we examine the scalability of Discus to large numbersof groups. Figure 10 shows that increasing the degree ofmembership for each node from2 to 1024 has almost noeffect on the percentage of packets recovered via LEC, andcauses a slow increase in average recovery latency. Notethat the x-axis in both these graphs is log-scale, and hencea straight line increase is actually logarithmic with respectto the number of groups, and represents excellent scalability.The increase in recovery latency towards the right side ofthe graph is due to Discus having to deal internally with therepresentation of large numbers of groups; we examine thisphenomenon later in this section.

For a comparison point, we refer readers back to SRM’sdiscovery time in Figure 2: in128 groups, SRM discoverytook place at 9 seconds. In our experiments, SRM recoverytook place roughly four seconds after discovery in all cases;we have not presented those results here, since we feel fine-tuning the SRM implementation for a clustered setting wouldbring recovery latency much closer to the lower bound ofthe discovery curve. However, the point remains that at128 groups we have already surpassed SRM’s best possiblerecovery performance by between two and three orders ofmagnitude.

11

0

20

40

60

80

100

0 0.05 0.1 0.15 0.2 0.25

LE

C R

ecov

ery

Perc

enta

ge

Loss Rate

Effect of Loss Rate on Recovery %

Bursty b=10Uniform

Markov m=10

10000 15000 20000 25000 30000 35000 40000 45000 50000 55000

0 0.05 0.1 0.15 0.2 0.25Rec

over

y L

aten

cy (M

icro

seco

nds)

Loss Rate

Effect of Loss Rate on Latency

Bursty b=10Uniform

Markov m=10

Figure 12: Impact of Loss Rate on LEC

Though Discus’s recovery characteristics scale well in thenumber of groups, it is important that the computational over-head imposed by the protocol on nodes stays manageable,given that time-critical applications are expected run over it.Figure 11 shows the scalability of an important metric: thetime taken to process a single data packet. The straight lineincrease against a log x-axis shows that per-packet process-ing time increases logarithmically with respect to the numberof groups - doubling the number of groups results in a con-stant increase in processing time. The increase in processingtime towards the latter half of the graph is due to the in-crease in the number of repair bins as the number of groupsincreases. While we did not address this problem in our cur-rent implementation - 1024 groups was considered adequatescalability - we could easily remove this scalability ‘hit’ bycreating bins only for occupied regions. With the currentimplementation of Discus, per-packet processing time goesfrom 160 microseconds for 2 groups to 300 microseconds for1024, supporting throughput exceeding a thousand packetsper second. Figure 11 also shows the average number ofXORs per incoming data packet. As stated in section 4.2.2,the number of XORs stays under 5 - the value ofc from therate-of-fire(r, c). Experiment Setup:(n = 64, d = ∗, s = 10),Loss Model: Uniform, 1%.

30

40

50

60

70

80

90

100

5 10 15 20 25 30 35 40

Rec

over

y Pe

rcen

tage

Burst Size

Resilience to Bursty Losses: Recovery Percentage

L=0.1%L=1%

L=10%

15000 20000 25000 30000 35000 40000 45000 50000 55000 60000 65000

5 10 15 20 25 30 35 40Rec

over

y L

aten

cy (M

icro

seco

nds)

Burst Size

Resilience to Bursty Losses: Recovery Latency

L=0.1%L=1%

L=10%

Figure 13: Resilience to Burstiness

5.4 Impact of Loss Rate on LEC EffectivenessFigure 12 shows the impact of the Loss Rate on LEC recov-ery characteristics, under the three loss models. Both LECrecovery percentages and latencies degrade gracefully: withan unrealistically high loss rate of 25%, Discus still recovers40% of lost packets at an average of 60 milliseconds. Notethat for uniform and bursty loss models, recovery percentagestays above 90% with a loss rate of 5%; markov does not fareas well, even at 1% loss rate, primarily because it inducesbursts much longer than its average of 10 - the max burstin this setting averages at 50 packets. Experiment Setup:(n = 64, d = 128, s = 10), Loss Model: *.

5.5 Resilience to Bursty LossesAs we noted before, a major criticism of FEC schemes is theirfragility in the face of bursty packet loss. Figure 13 showsthat Discus is naturally resilient to small loss bursts, withoutthe stagger optimization - however, as the burst size increases,the percentage of packets recovered using LEC degradessubstantially. Experiment Setup:(n = 64, d = 128, s = 10),Loss Model: Bursty.

However, switching on the stagger optimization describedin Section 4.2.2 increases Discus’s resilience to burstiness

12

10 20 30 40 50 60 70 80 90

100

1 2 3 4 5 6 7 8 9 10

Rec

over

y Pe

rcen

tage

Stagger

Staggering: LEC Recovery Percentage

b=10burst=50

burst=100Uniform

10000 15000 20000 25000 30000 35000 40000 45000 50000 55000 60000 65000

1 2 3 4 5 6 7 8 9 10Rec

over

y L

aten

cy (M

icro

seco

nds)

Stagger

Staggering: Recovery Latency

b=10burst=50

burst=100Uniform

Figure 14: Staggering

90

92

94

96

98

100

16 24 32 40 48 10000

20000

30000

40000

50000

60000

Rec

over

y Pe

rcen

tage

Rec

over

y L

aten

cy (m

icro

seco

nds)

Group Size

Recovery %Latency

Figure 15: Effect of Group Size

tremendously, without impacting recovery latency much.Figure 14 shows that setting an appropriate stagger valueallows Discus to handle large bursts of loss: for a burst sizeas large as100, a stagger of6 enables recovery of more than90% lost packets at an average latency of around 50 millisec-onds. Experiment Setup:(n = 64, d = 128, s = 10), LossModel: Bursty, 1%.

5.6 Effect of Group and System SizeWhat happens to LEC performance when the average groupsize in the cluster is large compared to the total number ofnodes? Figure 15 shows that recovery percentages are almostunaffected, staying above 99% in this scenario, but recoverylatency is impacted by more than a factor of 2 as we triplegroup size from 16 to 48 in a 64-node setting. Note thatthis measures the impact of the size of the group relativeto the entire system; receiver-based FEC has been shown toscale well in a single isolated group to hundreds of nodes [2].Experiment Setup:(n = 64, d = 128, s = ∗), Loss Model:Uniform, 1%.

While we were not able to evaluate to system sizes beyond64 nodes, Discus should be oblivious to the size of the entire

system, since each node is only concerned with the groups itbelongs to. We ran a single run of Discus with 4 instancesrunning on each node to obtain an emulated 256 node systemwith each instance in 128 groups, and the resulting recoverypercentage of 98% - albeit with a degraded average recov-ery latency of nearly 200 milliseconds due to network andCPU contention - confirmed our intuition of the protocol’sfundamental insensitivity to system size.

6. RELATED WORKReliable Multicast is a well-researched field, with a wealth ofexisting solutions - we have mentioned many of these in Sec-tion 3.1. Lightweight groups [12] attempt to scale strongermulticast properties like ordering and atomicity to greaternumbers of groups by multiplexing communication over asingle transport group - while this succeeds in amortizingthe overhead of maintaining these stronger properties acrossmultiple groups, nodes in the transport group are forced toreceive and discard irrelevant data from groups they do notbelong to. Data-Aware Multicast [1] optimizes the numberof groups in the system by using the hierarchical nature ofspecific topic-based publish-subscribe systems.

A substantial corpus of work exists in real-time distributedsystems - a categorization can be found in [18]. Many tradi-tional real-time protocols make assumptions about the exis-tence of time-aware components in the stack - such as real-time ethernet or operating systems - which are unrealistic ina COTS setting. Other work in this space offers real-time‘guarantees’ of meeting hard deadlines when the numberof faults in the system is under a threshold value, yieldingconservative protocols like CASD [7]. In contrast, Discusis aimed at applications requiring low-latency communica-tion that degrades gracefully in COTS environments, ratherthan absolute real-time guarantees; in a sense, it is closer tomodels such asquasi-synchronicity [18].

Another related body of work involves mathematically so-phisticated FEC encodings, such as Tornado [4] codes. Re-cently, the XOR has appeared as a useful primitive in wireless

13

network coding [14].

7. FUTURE WORKOne avenue of research involves embedding more complexerror codes such as Tornado [4] in LEC; however, the useof XOR has significant implications for the design of thealgorithm, and using a different encoding might require sig-nificant changes. LEC uses XOR for its simplicity and speed,and as our evaluation showed, we obtain properties on parwith more sophisticated encodings, including tunability andburst resilience. We plan on replacing our simplistic NAKlayer with a version optimized for bulk transfer, providingan efficient backup for Discus when sustained bursts occurof hundreds of packets or more. Another line of work in-volves making the parameters for LEC - such as rate-of-fireand stagger - adaptive, reacting to meet varying load andnetwork characteristics.

We are currently working with industry partners to layerDiscus under data distribution, publish-subscribe and web-service interfaces, as well as building protocols with strongerordering and atomicity properties over it. We are also de-veloping a run-time environment for time-critical replicatedservices which uses Discus as a fundamental communica-tion primitive. The Discus implementation is available fordownload (url removed for blinding purposes).

8. CONCLUSIONWe believe that the next generation of time-critical appli-cations will execute on commodity clusters, using the tech-niques of massive redundancy, fault-tolerance and scalablecommunication currently available to distributed systemspractitioners. Such applications will require a multicastprimitive that delivers data at the speed of hardware mul-ticast in failure-free operation and recovers from packet losswithin milliseconds irrespective of the pattern of usage. Dis-cus provides applications with massive scalability in multipledimensions - crucially, it scales in the number of groups inthe system, performing well under arbitrary grouping pat-terns and overlaps. A clustered communication primitivewith good timing properties can ultimately be of use to ap-plications in diverse domains not normally considered time-critical - e-tailers, online web-servers and enterprise applica-tions, to name a few.

9. ACKNOWLEDGMENTSRemoved for Blinding

10. REFERENCES[1] S. Baehni, P. T. Eugster, and R. Guerraoui. Data-aware

multicast. In Proceedings of the 2004 InternationalConference on Dependable Systems and Networks(DSN’04), page 233, Washington, DC, USA, 2004. IEEEComputer Society.

[2] M. Balakrishnan, S. Pleisch, and K. Birman. Slingshot:Time-critical multicast for clustered applications. In IEEENetwork Computing and Applications, 2005.

[3] K. P. Birman, M. Hayden, O. Ozkasap, Z. Xiao, M. Budiu,and Y. Minsky. Bimodal multicast. ACM Trans. Comput.Syst., 17(2):41–88, 1999.

[4] J. W. Byers, M. Luby, M. Mitzenmacher, and A. Rege. Adigital fountain approach to reliable distribution of bulkdata. In Proceedings of the ACM SIGCOMM ’98Conference, pages 56–67, New York, NY, USA, 1998. ACMPress.

[5] Y. Chawathe, S. McCanne, and E. A. Brewer. Rmx:Reliable multicast for heterogeneous networks. InINFOCOM, pages 795–804, 2000.

[6] Y. Chu, S. Rao, S. Seshan, and H. Zhang. Enablingconferencing applications on the internet using an overlaymuilticast architecture. In Proceedings of the ACMSIGCOMM ’01 Conference, pages 55–67, New York, NY,USA, 2001. ACM Press.

[7] F. Cristian, H. Aghali, R. Strong, and D. Dolev. Atomicbroadcast: From simple message diffusion to byzantineagreement. In Proc. 15th Int. Symp. on Fault-TolerantComputing (FTCS-15), pages 200–206, Ann Arbor, MI,USA, 1985. IEEE Computer Society Press.

[8] S. E. Deering and D. R. Cheriton. Multicast routing indatagram internetworks and extended lans. ACM Trans.Comput. Syst., 8(2):85–110, 1990.

[9] P. T. Eugster, R. Guerraoui, S. B. Handurukande,P. Kouznetsov, and A.-M. Kermarrec. Lightweightprobabilistic broadcast. ACM Trans. Comput. Syst.,21(4):341–374, 2003.

[10] S. Floyd, V. Jacobson, C.-G. Liu, S. McCanne, andL. Zhang. A reliable multicast framework for light-weightsessions and application level framing. IEEE/ACM Trans.Netw., 5(6):784–803, 1997.

[11] J. Gemmel, T. Montgomery, T. Speakman, N. Bhaskar,and J. Crowcroft. The pgm reliable multicast protocol.IEEE Network, 17(1):16–22, Jan 2003.

[12] K. Guo and L. Rodrigues. Dynamic light-weight groups. InProceedings of the 17th International Conference onDistributed Computing Systems (ICDCS ’97), page 33,Washington, DC, USA, 1997. IEEE Computer Society.

[13] C. Huitema. The case for packet level fec. In PfHSN ’96:Proceedings of the TC6 WG6.1/6.4 Fifth InternationalWorkshop on Protocols for High-Speed Networks V, pages109–120, London, UK, UK, 1997. Chapman & Hall, Ltd.

[14] S. Katti, D. Katabi, W. Hu, H. Rahul, and M. Medard.The importance of being opportunistic: Practical networkcoding for wireless environments. In 43rd Annual AllertonConference on Communication, Control and Computing,Sept 2005.

[15] J. C. Lin and S. Paul. RMTP: A reliable multicasttransport protocol. In INFOCOM, pages 1414–1424, SanFrancisco, CA, Mar. 1996.

[16] T. Montgomery, B. Whetten, M. Basavaiah, S. Paul,N. Rastogi, J. Conlan, and T. Yeh. The RMTP-II protocol,Apr. 1998. IETF Internet Draft.

[17] J. Nonnenmacher, E. Biersack, and D. Towsley.Parity-based loss recovery for reliable multicasttransmission. In Proceedings of the ACM SIGCOMM ’97conference, pages 289–300, New York, NY, USA, 1997.ACM Press.

[18] P. Verissimo and L. Rodrigues. Distributed Systems forSystem Architects. Kluwer Academic Publishers, Norwell,MA, USA, 2001.

[19] M. Yajnik, S. B. Moon, J. F. Kurose, and D. F. Towsley.Measurement and modeling of the temporal dependence inpacket loss. In INFOCOM, pages 345–352, 1999.

14