Search and Replication in Unstructured Peer-to-Peer

Networks

Pei Cao

Cisco Systems, Inc.

(Joint work with Christine Lv, Edith Cohen, Kai Li and Scott Shenker)

Disclaimer

• Results, statements, opinions in this talk do not represent Cisco in anyway

• This talk is about technical problems in networking, and does not discuss moral, legal and other issues related to P2P networks and their applications

Outline

• Brief survey of P2P architectures

• Evaluation methodologies

• Search methods

• Replication strategies and analysis

• Simulation results

Characteristics of Peer-to-Peer Networks

• Unregulated overlay network• Current application: file swapping• Dynamic: nodes join or leave frequently• Example systems:

– Napster, Gnutella;– Freenet, FreeHaven, MajoNation, Alpine, ...– JXTA, Ohaha, …– Chord, CAN, “Past”, “Tapestry”, Oceanstore

Architecture Comparisons

• Napster: centralized– A central website to hold file directory of all

participants; Very efficient– Scales– Problem: Single point of failure

• Gnutella: decentralized– No central directory; use “flooding w/ TTL”– Very resilient against failure– Problem: Doesn’t scale

Architecture Comparisons

• Various research projects such as CAN: decentralized, but “structured”– CAN: distributed hash table– “Structure”: all nodes participate in a precise

scheme to maintain certain invariants– Extra work when nodes join and leave– Scales very well, but can be fragile

Architecture Comparisons

• FreeNet: decentralized, but semi-structured– Intended for file storage– Files are stored along a route biased by hints– Queries for files follow a route biased by the

same hints– Scales very well– Problem: would it really work?

• Simulation says yes in most cases, but no proof so far

Our Focus: Gnutella-Style Systems

• Advantages of Gnutella: – Support more flexible queries

• Typically, precise “name” search is a small portion of all queries

– Simplicity, high resilience against node failures

• Problems of Gnutella: Scalability– Bottleneck: interrupt rates on individual nodes– Self-limiting network: nodes have to exit to get

real work done!

Evaluation Methodologies

Simulation based:

• Network topology

• Distribution of object popularity

• Distribution of replication density of objects

Evaluation Methods

• Network topologies:– Uniform Random Graph (Random)

• Average and median node degree is 4

– Power-Law Random Graph (PLRG)• max node degree: 1746, median: 1, average: 4.46

– Gnutella network snapshot (Gnutella)• Oct 2000 snapshot

• max degree: 136, median: 2, average: 5.5

– Two-dimensional grid (Grid)

Modeling Methods

• Object popularity distribution pi

– Uniform– Zipf-like

• Object replication density distribution ri

– Uniform

– Proportional: ri pi

– Square-Root: ri pi

Evaluation Metrics

• Overhead: average # of messages per node per query

• Probability of search success: Pr(success)

• Delay: # of hops till success

Load on Individual Nodes

• Why is a node interrupted:– To process a query– To route the query to other nodes– To process duplicated queries sent to it

Duplication in Flooding-Based Searches

. . . . . . . . . . . .

• Duplication increases as TTL increases in flooding

• Worst case: a node A is interrrupted by N * q * degree(A) messages

1

2 3 4

5 6 7 8

Duplications in Various Network Topologies

Flooding: % duplicate msgs vs TTL

0

20

40

60

80

100

2 3 4 5 6 7 8 9

TTL

du

pli

cate

msg

s (%

)

Random

PLRG

Gnutella

Grid

Relationship between TTL and Search Successes

Flooding: Pr(success) vs TTL

0

20

40

60

80

100

120

2 3 4 5 6 7 8 9

TTL

Pr(

succ

ess

) % Random

PLRG

Gnutella

Grid

Problems with Simple TTL-Based Flooding

• Hard to choose TTL:– For objects that are widely present in the

network, small TTLs suffice– For objects that are rare in the network, large

TTLs are necessary

• Number of query messages grow exponentially as TTL grows

Idea #1: Adaptively Adjust TTL

• “Expanding Ring”– Multiple floods: start with TTL=1; increment

TTL by 2 each time until search succeeds

• Success varies by network topology– For “Random”, 30- to 70- fold reduction in

message traffic– For Power-law and Gnutella graphs, only

3- to 9- fold reduction

Limitations of Expanding Ring

Flooding: #nodes visited vs TTL

0

2000

4000

6000

8000

10000

12000

2 3 4 5 6 7 8 9

TTL

#no

de

s vi

site

d

Random

PLRG

Gnutella

Grid

Idea #2: Random Walk

• Simple random walk– takes too long to find anything!

• Multiple-walker random walk– N agents after each walking T steps visits as

many nodes as 1 agent walking N*T steps– When to terminate the search: check back with

the query originator once every C steps

Search Traffic Comparison

avg. # msgs per node per query

1.863

2.85

0.053

0.961

0.027 0.0310

0.5

1

1.5

2

2.5

3

Random Gnutella

Flood Ring Walk

Search Delay Comparison

# hops till success

2.51 2.39

4.033.4

9.12

7.3

0

2

4

6

8

10

Random Gnutella

Flood Ring Walk

Lessons Learnt about Search Methods

• Adaptive termination

• Minimize message duplication

• Small expansion in each step

Flexible Replication

• In unstructured systems, search success is essentially about coverage: visiting enough nodes to probabilistically find the object => replication density matters

• Limited node storage => what’s the optimal replication density distribution?– In Gnutella, only nodes who query an object store it =>

ri pi

– What if we have different replication strategies?

Optimal ri Distribution

• Goal: minimize ( pi/ ri ), where ri =R

• Calculation: – introduce Lagrange multiplier , find ri and

that minimize:

( pi/ ri ) + * ( ri - R)

=> - pi/ ri2 = 0 for all i

=> ri pi

Square-Root Distribution

• General principle: to minimize ( pi/ ri ) under constraint ri =R, make ri propotional to square root of pi

• Other application examples:– Bandwidth allocation to minimize expected

download times– Server load balancing to minimize expected

request latency

Achieving Square-Root Distribution

• Suggestions from some heuristics– Store an object at a number of nodes that is

proportional to the number of node visited in order to find the object

– Each node uses random replacement

• Two implementations:– Path replication: store the object along the path of a

successful “walk”– Random replication: store the object randomly among

nodes visited by the agents

Evaluation of Replication Methods

• Metrics– Overall message traffic– Search delay

• Dynamic simulation– Assume Zipf-like object query probability– 5 query/sec Poisson arrival– Results are during 5000sec-9000sec

Distribution of ri

Replication Distribution: Path Replication

0.001

0.01

0.1

1

1 10 100

object rank

rep

lica

tio

n r

ati

o

(no

rma

lize

d)

real result

square root

Total Search Message Comparison

• Observation: path replication is slightly inferior to random replication

Avg. # msgs per node (5000-9000sec)

0

10000

20000

30000

40000

50000

60000

Owner Rep

Path Rep

Random Rep

Search Delay Comparison

Dynamic simulation: Hop Distribution (5000~9000s)

0

20

40

60

80

100

120

1 2 4 8 16 32 64 128 256

#hops

qu

eri

es

fin

ish

ed

(%

)

Owner Replication

Path Replication

Random Replication

Summary

• Multi-walker random walk scales much better than flooding– It won’t scale as perfectly as structured

network, but current unstructured network can be improved significantly

• Square-root replication distribution is desirable and can be achieved via path replication