The bandwidth estimation by the visualization of TCP...
-
Upload
vuongkhuong -
Category
Documents
-
view
217 -
download
3
Transcript of The bandwidth estimation by the visualization of TCP...
![Page 1: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/1.jpg)
The bandwidth estimation by the visualization of TCP
window sizeS.Y.Suzuki, H.Matsunaga (KEK)
1
![Page 2: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/2.jpg)
• perfSONAR is widely used in the LHC universe, to monitor the network stability.
• It periodically runs bwctl and owamp tests to multiple sites and records the result in the database.- bwctl internally uses iperf or nuttcp
• and shows histories graphically.
bandwidth historyone way latency history
2
![Page 3: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/3.jpg)
Unfortunately, KEK is not joining to LHC tier sites.
Recently, the computing activity in HEP is led by LHC.
KEK
LHC
3
![Page 4: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/4.jpg)
• Major reasons- End nodes are not tuned for the long latency transmission.
• Easy to solve. Just tune the system parameter.
- Intermediate link is congested.
• Difficult to solve.
• The congested link is shared by several activities.
• Sometimes- Remote firewall dislike KEK.
• Firewall is NOT always operated by physicists and KEK is still nameless for non-HEP people.
• FW admin "Our firewall already permits major HEP labs for LHC!KEK? what is that?".
Users voice: "Hi, the transfer from KEK to my institute is very slow. Why?"
4
![Page 5: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/5.jpg)
Response for them
• If their institute runs already perfSONAR, that is quite useful.- "Just temporal congestion. Please try another day."- "It is always saturated. Please express upgrade necessity to your upstream
network!"
• perfSONAR contains NPAD and NDT for the diagnostic of the end nodes.- NDT measures the bandwidth- NPAD checks more details and
reports several problems if exist.
5
![Page 6: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/6.jpg)
Bandwidth report of perfSONAR
• The bandwidth report of perfSONAR is made by iperf or nuttcp.
• The report value is an average speed during 20 or 30 seconds.
• If there is a packet loss, the average value is bad.• In the case of long RTT, the average value becomes worse.
Test endTest start
Spee
d
AverageActu
al s
peed
Maximum speed
Test endTest start
Spee
d
AverageActual speed
Maximum speedShort RTT Long RTT(If no packet loss)
6
![Page 7: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/7.jpg)
• TCP is very modest and cooperative protocol.
• When TCP found a congestion, it reduces the speed pessimistically.
• Recovery speed depends on the RTT.
• Once packet loss has happened, always TCP on long RTT loses the race.
The TCP behavior
A B C Dloss
Transmission start
Spee
d loss loss loss loss
loss loss loss loss loss loss loss loss
TimeTransmission start
Spee
d Available bandwidth
TimeTransmission start
Spee
d Available bandwidthloss loss loss
short RTT
long RTT
7
![Page 8: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/8.jpg)
• tcp_probe
- requires a root privilege to insert kernel module for the sender machine.
• wireshark
- uses only information in ACK packets from the receiver, so the congestion window size will not be shown.
• http://ask.wireshark.org/questions/2820/how-to-get-cwnd-in-wireshark
This behavior can be visualized by
8
![Page 9: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/9.jpg)
What is thecongestion window?
• Receiver sides announces "the limit you can send"- in every ACK packet.
• Sender CAN send within the limit, but actually sends slower than that in order to avoid the congestion.- This is the "congestion window size" and it internally exists in the sender kernel.
• The actual speed depends on this congestion window size.
• If packets are lost, the congestion window size shrinks.
Time
Advertized window size
Con
gest
ion
win
dow
siz
e
loss loss loss loss
Transmission start
9
![Page 10: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/10.jpg)
How to detect the congestion window size?
• peep directly kernel- tcp_probe
• Guess from the packet sequence at the sender side.- Using absolute time information
of captured packets.- We use this way
Time ACK
PSH
Data receiverData sender
Adve
rtise
d w
indo
w s
ize
cong
estio
n w
indo
w s
ize
As the congestion window determines the actual speed,the recording of its behavior is a precise speed measurement.
10
![Page 11: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/11.jpg)
• For LHC guys- the grid-ftp using multi TCP streams
• multiple TCP enables the high-speed transfer with the small window size.
- Run multiple grid-ftps in non-interactively.
• Never mind, your transmission will be finished in someday.
- Bandwidth allocation if possible.
• Not always applicable, especially non-LHC sites.- including KEK
- Data in KEK is not always exportable via GRID
• Some of them are exportable, but some of them are not.
For the long distance data transmission
11
![Page 12: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/12.jpg)
Case 1.Nagoya univ. and KEK
12
![Page 13: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/13.jpg)
• From KEK to Nagoya is about 400km
Nagoya-U is the most powerful and nearest Belle collaborator.
KEK
Nagoya
13
![Page 14: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/14.jpg)
perfSONAR shows a very jaggy bandwidth history
What is happening?
14
![Page 15: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/15.jpg)
Sometimes good, but sometimes bad.But most of the packet loss happen over 900Mbps.
Observed behaviors in each data point.
980Mbps in average
200Mbps in average
15
![Page 16: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/16.jpg)
After the QoS limitation up to 925Mbps
But we do not recommend QoS on the regular tests of perfSONAR,because it breaks the comparability of trends to other sites.
Possibly other activities use about 60-100Mbps
16
![Page 17: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/17.jpg)
Case 2.DESY and KEK
17
![Page 18: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/18.jpg)
• that is, "slow"
• At 2009, ILC collaborator in DESY questioned about similar problem.
- At that time, the problem is just window size configuration.
- After proper setting, 900Mbps was achieved.
- QoS is also very effective.
• Now DESY is running perfSONAR server for LHC,but it was not opened to KEK at that time.
- Instead of that, DESY collaborator opens iperf server immediately. We can test the speed by that.
Last year we asked from Belle collaborator in DESY.
18
![Page 19: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/19.jpg)
At 2009, 900Mbps is achieved 24MB wsize
19
![Page 20: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/20.jpg)
But 1Gbs was hard to use
There is a significant interference between reversal connection.We applied QoS to reduce the maximum speed to 900Mbps, then the interference seemed to be away.
Bi-directional test between DESY and KEK
KEK->DESY
DESY->KEK
20
![Page 21: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/21.jpg)
• A problem of the end-node, not of the network.
• Sometimes the window size is very smaller than 3.4MB.
• Is there any time dependency?
- If perfSONAR had been available at that time, it is very easy question.
• It maybe hard to get a permission to access perfSONAR soon, so we did DIY.
In this time, the window size is limited 3.4MB.
100kB
3.4MB
Better day
worse day
21
![Page 22: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/22.jpg)
Script for periodical test#!/bin/shcd $HOME/desysuffix=`date +%Y.%m%d.%H%M`exec > log.$suffixexec 2>&1tcpdump -i eth0 -n -w pkts.$suffix -p port 33200 &tcpdump_pid=$!sleep 5iperf -c ******************* -t 10 -w 32M -i 2 -p ******kill $tcpdump_pid
test by iperf and capture the packetsto make graph of the window size
22
![Page 23: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/23.jpg)
Run it in every 10min.
23
![Page 24: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/24.jpg)
Plot these graphsas a 2D histogram
Time since the iperf begins
(sec)
When the iperf is started
(Date)
Win
dow
siz
e
24
![Page 25: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/25.jpg)
A few days history
Spikes caused by software bug,please ignore.
25
![Page 26: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/26.jpg)
Shapes of good and bad
Bad
Good
Bad condition makes “tail”26
![Page 27: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/27.jpg)
The top view of the histogram
Rare bads
Bads probability is not uniform.
Many bads
27
![Page 28: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/28.jpg)
Activity of weekday?
Weekend
Weekend
iperf test duration was extended to 20sec from
10sec
28
![Page 29: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/29.jpg)
In weekend, we can extend the window size.• From the 2009 test, the window size should be about
25MB for 1Gbps from KEK to DESY.
• DESY collaborator extended it for this test.
• But the window size > 18MB makes heavy packet loss.
Packet loss
Extend
3.4MB
29
![Page 30: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/30.jpg)
Result of Nov. 2012weekend
weekend
weekend
30
![Page 31: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/31.jpg)
DESY permitted perfSONAR access from KEK
KEK->DESYDESY->KEK Is available bandwidth 250Mbps...?
More faster!
31
![Page 32: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/32.jpg)
Speed improvement
~10MB/s
45-55MB/s (500-600Mbps)
Extending window size...
32
![Page 33: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/33.jpg)
Nov.2012 - Jan.2013
Nov. 2012
Jan. 2013
Xmas+new year
33
![Page 34: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/34.jpg)
Even in holidays,
34
![Page 35: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/35.jpg)
Available bandwidth is about 60MB/s
35
![Page 36: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/36.jpg)
The best case
Window size reached 20MB,
Achieved speed is 70MB/s.But this is very rare.
Window size
Achieved speed
36
![Page 37: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/37.jpg)
Summary
• Although perfSONAR is very effective to find a trend of the bandwidth, but it is not always available.
• The detail observation of TCP congestion window is useful to estimate the available bandwidth even in such a case.- If we can precisely estimate it, the parameter tuning and QoS
application will make better performance.
37
![Page 38: The bandwidth estimation by the visualization of TCP ...ssrc.inp.nsk.su/conf/AFAD2013/presentations/WG6/5... · •perfSONAR is widely used in the LHC universe, to monitor the network](https://reader034.fdocuments.us/reader034/viewer/2022051718/5a71c0f67f8b9abb538d0e59/html5/thumbnails/38.jpg)
Special thanks to
• Kiyoshi Hayasaka (Nagoya University)
• Andreas Gellrich (DESY)
• Kind coordination for perfSONAR servers and an iperf server in remote site.
38