On-the-fly Verification of Erasure-Encoded

File Transfers

Mike Freedman & Max KrohnNYU Dept of Computer Science

Downloading Large Files From P2P Networks For large files, transfer times are much

bigger than average node uptimes.

Some files are very popular: multiple sources and multiple requesting nodes.

Is it possible to have multicast, even though sources and receivers frequently enter and leave the network.

Solution: Rateless Erasure Codes

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Solution: Rateless Erasure Codes

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

( )h F

( )h F ( )h F( )h F

Wants file F

Mutli-Sourced Downloads

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Mutli-Sourced Downloads

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Receiver (R3) Receiver (R4)

Receiver (R3)

“Overlapping Multicast Trees”

Source (S1) Source (S2) Source (S3) Source (S4)

Receiver (R2)

Receiver (R1)

Resuming Truncated Downloads

Source (S1)

Receiver (R1) Receiver (R2)

Resuming Truncated Downloads

Source (S1)

Receiver (R1) Receiver (R2)

Resuming Truncated Downloads

Source (S1)

Receiver (R1) Receiver (R2)

Threat Model

KaZaaKaZaa

eDonkey 2000

Gnutella

Morpheus

Threat Model

KaZaaKaZaaeDonkey 2000

Gnutella

Morpheus

Threat Model

KaZaaKaZaaeDonkey 2000

Gnutella

Morpheus

Threat Model

KaZaaKaZaa

eDonkey 2000

Gnutella

Morpheus

Bogus Data Attack

KaZaaKaZaaeDonkey 2000

Gnutella

Morpheus

Unwanted Data Attack

KaZaaKaZaaeDonkey 2000

Gnutella

Morpheus

Attacking Erasure Encoded Transfers

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Attacking Erasure Encoded Transfers

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Erasure Encoding of Files

1b 2b 3b 4b

1c 2c 3c 4c 5c 6c

1 2 3

2 2 3 4

c b b

c b b b

3 1 4

4 3 4

c b b

c b b

F

…

Easily Verifiable….

1( ) ( ( ),..., ( ))nh F h b h b

1b 2b 3b 4b

1c 2c 3c 4c 5c 6c

F

…

…but Not on the Fly

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

What Happened?R1 received checkblock c from

S4. S4 claims:

blah

9813024 bbbc

What Happened?R1 received checkblock c from

S4. S4 claims:R1 knows:

But how can R1 verify c?

Wouldn’t it be nice if:

Not true for SHA1!

9813024 bbbc )(),(),( 9813024 bhbhbh

What Happened?R1 received checkblock c from

S4. S4 claims:R1 knows: But how can R1 verify c?

Wouldn’t it be nice if:

Not true for SHA1!

9813024 bbbc )(),(),( 9813024 bhbhbh

What Happened?R1 received checkblock c from

S4. S4 claims:R1 knows: But how can R1 verify c?Wouldn’t it be nice if:

Not true for SHA1!

9813024 bbbc )(),(),( 9813024 bhbhbh

)()()()( 9813024 bhbhbhch

What Happened?R1 received checkblock c from

S4. S4 claims:R1 knows: But how can R1 verify c?Wouldn’t it be nice if:

Not true for SHA1!

9813024 bbbc )(),(),( 9813024 bhbhbh

)()()()( 9813024 bhbhbhch

A Homomorphic Hashing Scheme Assume file block size of 8kB

Pick large prime (about 1024 bits) and small prime (about 256 bits) that divides , and 256 generators of order q:

Writes the file F as matrix, elements in

pq

1,1 1,

1

256,1 256,

( )n

n

n

b b

b b

F b b

( 1)p

1 256( ,..., )g gg

qZ

How To Hash The hash of a message or check block is

an element in :qZ

1,1

2,1

1,1 2,1 256,1

256,1

1

11,1

2,1 21 2 256

256,1256

( )

( )( ) ( )x

b

bb b b

b

h

gb

b gg g g

b g

g

b

Π

How To Hash The hash of a message or check block is

an element in :

The hash of the entire file is an n-element

vector of the hashes of the blocks:

1( ) ( ( ),..., ( ))nH h h h F b b

,

256

1

( ) (mod )i kbk i

i

h g p

b

qZ

The Only Important Slide

( ) ( ) ( )j k j kh h h b b b b

,

256

1

( ) (mod )i kbk i

i

h g p

b

implies that

Why?

a b a bg g g

How To Encode Checkblocks are constructed using

modular addition over .

To generate a checkblock, pick a set

{1,..., }S n

(mod )S kk S

q

c b

qZ

And compute

How To VerifyGiven the correct hash:

And a check block:

verify that:

Note: LHS computation is expensive!

1( ( ),..., ( ))nH h h b b

( ) ( )S kk S

h h

c b

(mod )S kk S

q

c b

Success!

Source (S1)

Receiver (R1)

Source (S2) Source (S3) Source (S4)

Analysis+ Security of the hash function based on

hardness of the discrete log.

− Hashes are big (1/256 the size of the file), but we can apply this process recursively.

+ Our paper details a batched, probabilistic verification scheme that drastically reduces exponentiations.

+ Verifying rate is 40x faster than download rates on a T1.