High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking...
-
date post
20-Dec-2015 -
Category
Documents
-
view
217 -
download
3
Transcript of High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking...
![Page 1: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/1.jpg)
High Performance Networking with
Little or No Buffers
Yashar Ganjali
High Performance Networking GroupStanford University
[email protected]://www.stanford.edu/~yganjali
Joint work with:Guido Appenzeller, Ashish Goel, Tim Roughgarden, Nick McKeown
May 5, 2005
![Page 2: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/2.jpg)
May 5, 2005 2
MotivationNetworks with Little or No
Buffers Problem
Internet traffic is doubled every year Disparity between traffic and router
growth (space, power, cost) Possible Solution
All-Optical Networking Consequences
Large capacity large traffic Little or no buffers
![Page 3: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/3.jpg)
May 5, 2005 3
Which would you choose?
DSL Router 1 DSL Router 2
$504 x 10/100 Ethernet
1.5Mb/s DSL connection
1Mbit of packet buffer
$554 x 10/100 Ethernet
1.5Mb/s DSL connection
4Mbit of packet buffer
Bigger buffers are better
![Page 4: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/4.jpg)
May 5, 2005 4
What we learn in school
Packet switching is good Long haul links are expensive Statistical multiplexing allows efficient
sharing of long haul links Packet switching requires buffers Packet loss is bad Use big buffers Luckily, big buffers are cheap
![Page 5: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/5.jpg)
May 5, 2005 5
Statistical Multiplexing
Observations1. The bigger the buffer, the lower the packet loss.2. If the buffer never goes empty, the outgoing line
is busy 100% of the time.
![Page 6: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/6.jpg)
May 5, 2005 6
What we learn in school
Queueing Theory
kkXP
EX
1
11
M/M/1
X
Buffer size
Loss rate
Observations1. Can pick buffer size for a given loss rate.2. Loss rate falls fast with increasing buffer size.3. Bigger is better.
![Page 7: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/7.jpg)
May 5, 2005 7
What we learn in school
Moore’s Law: Memory is plentiful and halves in price every 18 months. 1Gbit memory holds 500k packets
and costs $25.
Conclusion: Make buffers big. Choose the $55 DSL router.
![Page 8: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/8.jpg)
May 5, 2005 8
Why bigger isn’t better
Network users don’t like buffers Network operators don’t like buffers Router architects don’t like buffers We don’t need big buffers We’d often be better off with smaller
ones
![Page 9: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/9.jpg)
May 5, 2005 9
Backbone Router Buffers
Universally applied rule-of-thumb: A router needs a buffer size:
• 2T is the two-way propagation delay• C is capacity of bottleneck line
Context Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines.. Usually referenced to Villamizar and Song: “High
Performance TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design
CTB 2
CRouterSource Destination
2T
![Page 10: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/10.jpg)
May 5, 2005 10
Review: TCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2
![Page 11: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/11.jpg)
May 5, 2005 11
Review: TCP Congestion Control
Only W packets may be outstanding
Rule for adjusting W If an ACK is received: W ← W+1/W If a packet is lost: W ← W/2
Source Dest
maxW
2maxW
t
Window size
![Page 12: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/12.jpg)
May 5, 2005 12
Buffer Size in the Core
ProbabilityDistribution
B
0
Buffer Size
W
![Page 13: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/13.jpg)
May 5, 2005 13
Backbone router buffers
It turns out that The rule of thumb is wrong for a core
routers today
Required buffer is instead of CT 2n
CT 2
![Page 14: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/14.jpg)
May 5, 2005 14
2T C
n
Simulation
Required Buffer Size
![Page 15: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/15.jpg)
May 5, 2005 15
Validation
Theoretical results validated by: Thousands of ns2 simulations Network lab (Cisco routers) at University
of Wisconsin Stanford University dorm traffic Internet2 experiments
Ongoing work with network operators and router vendors…
![Page 16: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/16.jpg)
May 5, 2005 16
Impact on Router Design
10Gb/s linecard with 200,000 x 56kb/s flows Rule-of-thumb: Buffer = 2.5Gbits
• Requires external, slow DRAM Becomes: Buffer = 6Mbits
• Can use on-chip, fast SRAM• Completion time halved for short-flows
40Gb/s linecard with 40,000 x 1Mb/s flows Rule-of-thumb: Buffer = 10Gbits Becomes: Buffer = 50Mbits
![Page 17: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/17.jpg)
May 5, 2005 17
How small can buffers be?
Imagine you want to build an all-optical router for a backbone network…
…and you can build a few dozen packets in delay lines.
Conventional wisdom: It’s a routing problem (hence deflection routing, burst-switching, etc.)
Our belief: First, think about congestion control.
![Page 18: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/18.jpg)
May 5, 2005 18
TCP with ALMOST No Buffers
Utilization of bottleneck link = 75%
![Page 19: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/19.jpg)
May 5, 2005 19
Two Concurrent TCP Flows
![Page 20: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/20.jpg)
May 5, 2005 20
TCP Throughput with Small Buffers
TCP Throughput vs. Number of Flows
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 200 400 600 800 1000Number of Flows
Th
rou
gh
pu
t
![Page 21: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/21.jpg)
May 5, 2005 21
TCP Reno Performance
Buffer Size = 10; Load = 80%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1000 2000 3000 4000 5000 6000
Bottleneck Capacity Mbps
Th
rou
gh
pu
t
![Page 22: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/22.jpg)
May 5, 2005 22
The chasm between theory and practice
M/M/1
11
X
kkXP
EX
1
= 50%, EX = 1 packet = 75%, EX = 3 packets
= 50%, P[X>10] < 10-3
= 75%, P[X>10] < 0.06
Theory (benign conditions) Practice
Typical OC192 router linecard buffers over 2,000,000 packets
Can we make the traffic arriving at the routersPoisson “enough” to get most of the benefit?
![Page 23: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/23.jpg)
May 5, 2005 23
Ideal Solution
If packets are spaced out perfectly; and The starting times of flows are chosen
randomly; We only need a small buffer for contention
resolution.
![Page 24: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/24.jpg)
May 5, 2005 24
Pacing We need to break bursts
Modify TCP: Instead of sending packets when your receive ACKS send packets with a fixed rate of CWND/RTT.
Rely on network properties:• Access links throttle the flows to low rate• Core:Acess > 1000:1• TCP’s window size is limited today.
If these properties make the flow look like poisson with only 5-10 packets of buffering we can get 70-80% throughput.
![Page 25: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/25.jpg)
May 5, 2005 25
What we know so farabout very small buffers
ArbitraryInjectionProcess
If Poisson Processwith load < 1
CompleteCentralized
Control
Any rate > 0need unbounded
buffers
Theory Experiment
Need buffersize of approx:
O(logD + logW)i.e. 20-30 pkts
D=#of hopsW=window size
TCP Pacing:Results as goodor better than forPoisson
Constant fractionthroughput withconstant buffers
[Leighton 1999]
![Page 26: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/26.jpg)
May 5, 2005 26
CWND: Reno vs. Paced TCP
![Page 27: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/27.jpg)
May 5, 2005 27
TCP Reno: Throughput vs. Buffer Size
![Page 28: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/28.jpg)
May 5, 2005 28
Paced TCP: Throughput vs. Buffer Size
![Page 29: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/29.jpg)
May 5, 2005 29
Early resultsCongested core router with 10 packet buffers.Average offered load = 80%RTT = 100ms; each flow limited to 2.5Mb/s
router
source
server
source
10Gb/s
>10Gb/s >10Gb/s
![Page 30: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/30.jpg)
May 5, 2005 30
Slow access links, lots of flows, 10 packet buffers
router
source
server
source
10Gb/s
5Mb/s 5Mb/s
Congested core router with 10 packet buffers.RTT = 100ms; each flow limited to 2.5Mb/s
![Page 31: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/31.jpg)
May 5, 2005 31
Conclusion
We can reduce 1,000,000 packet buffers to 10,000 today.
We can probably reduce to 10-20 packet buffers: With many small flows, no change needed With some large flows, need pacing in the
access routers or at the edge devices. Need more experiments.
![Page 32: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/32.jpg)
May 5, 2005 32
Experiments
Performance measurement with Small (thousands of packets); and Tiny (tens of packets) buffers
Metrics: Link utilization (goodput/throughput) Drops Buffer occupancy Etc.
Data Gathered for minutes to days High load (50-70% utilization) is better
![Page 33: High Performance Networking with Little or No Buffers Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu yganjali.](https://reader036.fdocuments.us/reader036/viewer/2022062516/56649d415503460f94a1ca5d/html5/thumbnails/33.jpg)
Thank you!Thank you!
Questions?Questions?