ICNP’09, Princeton, NJ

CUBS: Coordinated Upload Bandwidth Sharing in Residential Networks

Enhua Tan1, Lei Guo2, Songqing Chen3, Xiaodong Zhang1

1The Ohio State University2Yahoo! Inc

3George Mason University

2

Residential Networks: Broadband Everywhere

60% in 2005

93% in 2009

3

Residential Networks Bandwidth• Residential networks download / upload bandwidth :

asymmetric -- lower upload bandwidth:– Typical DSL: 384 Kbps upload, 1.5 Mbps download

• Affected by line quality and distance– Typical Cable: 512 Kbps upload, 7 Mbps download

– Cable Network Bandwidth Allocation in U.S. :

4

Advantages of Limited Upload Bandwidth

• Most home users mainly utilize download bandwidth

• Residential network physical infrastructure fits limited upload bandwidth:– Cable network: tree structure

• Asymmetric bandwidth less attractive to business users:– Servers pay more because of upload traffic (symmetric)

5

Problems Caused by Limited Upload Bandwidth

• Increasingly demanding and dependable P2P-based applications demand equal upload to download bandwidth in principle– Large file download: slow upload peer suffers from

slow download due to tit-for-tat incentive– VoIP: demands substantial and stable upload

bandwidth to avoid jitters– Internet live streaming: slow upload leads to

quality degradation

6

Case 1: VoIP Performance Deteriorates

274 ms jitter!Ping RTT increased

by about 1,000 ms!

• Our Internet experiments: computer connects to cable modem

• Upload traffic affects ping RTT (round-trip time) from 28ms to 1,000 ms or around:– VoIP demands < 150ms one-way delay: VoIP works poorly!

7

Case 2: BitTorrent Downloads Slowly• BitTorrent tit-for-

tat: download performance constrained by the cap of upload bandwidth– Our Internet

experiments show this effect

Download time increases when upload cap decreases

8

Would Upload Bandwidth Increase Effectively Address the Problems?

• Observed 25 residential networks for 21 days, we had two main findings:– Upload channel utilization is bursty and unbalanced:

< 20% users highly active in cable network– Overall, about 50% - 80% upload bandwidth is idle

Simply increasing the upload bandwidth is not a cost-effective solution: more idle upload bandwidth would be underutilized

9

Our Solution –CUBS: Coordinated Upload Bandwidth Sharing

• CUBS monitors availability of upload bandwidth from all members

• Utilize idle upload bandwidth from neighboring cable or DSL networks

• Bandwidth sharing is fair and flexible

Significantly improve performance of upload intensive applications without affecting the performance of neighboring networks

10

Outline

• Problem Statement and Proposal• Internet Measurements• System Design• Evaluation• Conclusion

11

Residential Upload Bandwidth Measurement

• Collected IP addresses using Gnutella crawler• Identified Top-7 ISPs: Charter, Comcast, Cox, Road

Runner, Ameritech (AT&T), Pacbell, and Verizon• Scanned 25 subnets by probing from Planet-Lab:

up to 78.8% subnet IPs respond– 25 subnets with 2,040 IP addresses for 21 days (during

February and March, 2008)– no control on the residential hosts: ICMP ping probing

to estimate upload bandwidth– slightly underestimate the available bandwidth: on the

conservative side

12

Measured Results

• Cable networks have higher upload capacity than DSL

• Average Idle upload bandwidth ratio: Cable higher than DSL

Cable

DSL

62% -

93%52%

- 83%

13

Available Upload Bandwidth Distribution

< 10% hosts use more than half of

upload bandwidth:Unbalanced

DSL upload bandwidth varies among hosts (due

to varied line length)

Cable

DSL

Varies little along time:

Chance to utilize

14

Feasibilities of CUBSOur measurements show that:• In broadband networks, there exists plenty of unused

upload bandwidth: 50% - 80%:– Suggests that CUBS is feasible to utilize the

unused bandwidth

• Upload channel is utilized in an unbalanced way (less than 20% users fully utilized their upload bandwidth):– CUBS will be feasible to increase upload

bandwidth without demanding additional bandwidth supply

15

Outline

• Problem Statement and Proposal• Internet Measurements• System Design• Evaluation• Conclusion

16

Challenges of CUBS Design

• How to share upload bandwidth?

• How to discover unused upload bandwidth?

• How to encourage users to participate?

17

Residential Networks: Wi-Fi Everywhere

• Jupiter Research: 65% of the U.S. households use Wireless connections in 2006 -- higher usage now

• A study in 2006: # of APs (Access Points) detected in residential areas ranges 2-20

Residential Network

Cable/DSL Modem

Wireless Access Point (AP)

Wireless Station

18

CUBS System

Station A

Station B

Station C

Local AP of A

Local AP of B

Local AP of C

Periodically estimate local AP’s idle upload bandwidth

CUBS System

Associate with foreign AP

Active using upload bandwidth now

Associate with another foreign AP

19

CUBS Overlay: Discover Bandwidth

• Bootstrap: a tracker to record the local AP-ID and IP address of each node– The local AP-ID: {ESSID, AP MAC address}

• Overlay connections: new node queries tracker to connect with nearby nodes

• Idle bandwidth discovery: each node measures its local AP’s upload bandwidth usage to compute idle bandwidth, and shares this info on overlay

20

CUBS Node: Wireless for Sharing• Dual connections: branching node’s upload traffic to– the local AP through one network interface– the foreign AP through another wireless interface

• Choose foreign AP for bandwidth sharing:– highest available upload bandwidth– acceptable signal strength

• Connection binding (network layer): – modifies the connect system call – binds new connection to one of the available network

interfaces based on traffic load (or randomly)

21

Fair Usage of the Upload Bandwidth

• Serves as an incentive mechanism to encourage users to share bandwidth over CUBS

• Foreign AP owner is able to fully use upload bandwidth:– Tell other users to refrain from competitions when

needed– Other user restrains further usage of the foreign AP,

and tries to hand-off to another foreign AP

22

No Excessive Usage of Download

• For a new TCP connection:– If it is upload intensive or bidirectional, the

connection can be bound to the foreign AP– Otherwise should be bound to local AP

• Requires prediction of a connection’s upload to download traffic ratio

23

No Excessive Usage of Download: Prediction Evaluation

• Trace-driven evaluations:– packet header trace

of Dartmouth residential wireless network (2004)

– uses the mean of the upload-to-download traffic ratios of the latest n connections to the same host for prediction

for more than 80% connections, the prediction difference is less than 20%

24

Discussions

• Overlay-based user authentication:– AP owner controls who can use the AP through AP

web service to prevents abusive usage

• Multiplexing one wireless interface:– FatVAP (NSDI’08) allows one wireless interface to

associate with multiple APs: utilize Power-Saving Mode buffering

– Can be useful for CUBS user with only one interface

25

Discussions (Cont.)

• Mesh Networks for Sharing Long Distance APs:– CUBS node currently only connects to neighboring

APs to harness available bandwidth– CUBS wireless mesh network can route an upload

packet to the Internet via an AP which can be out of the signal range of the originator

– Then upload bandwidth can be shared among any nodes in the mesh network

26

Outline

• Problem Statement and Proposal• Internet Measurements• System Design• Evaluation• Conclusion

27

Evaluation Setup

• To emulate the residential network, we use NISTNet to control the Internet connections of the APs

• Configure the upload and download capacity to be 500 Kbps and 7 Mbps – typical bandwidth caps for a cable subscription

28

BitTorrent Experiments on Planet-Lab• Repeatable swarm: 1 seed and 18 peers on Planet-

Lab• Download lasts for one hour, w/o and w/ CUBS

• The average download throughput is improved by > 30%:– No matter it is load-based or hash-based (remote IP)

29

Fair Bandwidth Sharing in CUBSAP owner gets full upload bandwidth

quickly

Idle bandwidth is detected and utilized again

Utilize foreign AP’s upload bandwidth

30

Conclusion• A low upload bandwidth cap in residential networks causes

problems to P2P applications

• Measurements show the total upload is often underutilized– And the upload channel utilization is bursty and unbalanced

• CUBS enables a user to share idle upload bandwidth of its coordinated neighbors:– Improve the performance of delay-sensitive or P2P applications– Without demanding additional bandwidth supply from ISPs or

infrastructure support– Flexibly shares idle upload bandwidth while providing fairness

guarantees

31

Thank you!

Enhua Tan: etan@cse.ohio-state.eduhttp://www.cse.ohio-state.edu/hpcs/

32

Probing Ping RTT Distribution (When Upload Channel is Saturated)

• most RTTs are larger than 500ms:– implies that

queue lengths of residential networks are very large

– VoIP quality will degrade when upload is saturated