TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.
-
Upload
michael-potter -
Category
Documents
-
view
212 -
download
0
Transcript of TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.
![Page 1: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/1.jpg)
TCP Performance over Gigabit Networks
Arne Øslebø
NORDUnet2002
![Page 2: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/2.jpg)
Modern Network Interface Cards
● Programmable network processors● Hardware checksuming
● Offloads main CPU● Interrupt coalescing
● Less interrupts● On board memory● Zero-copy
● Mainly operating system issue
![Page 3: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/3.jpg)
Test tools● netperf
– used for measuring throughput between two systems.– http://www.netperf.org/
● vmstat / Microsoft Management Console– gives information about CPU load and interrupts/s.
● netstat– gives information about packet loss, retransmitions etc.
● One testrun consists of:– running netperf using various TCP window sizes.– for each TCP window size, netperf is run three or more
times. Results presented are average values.– Before running netperf, information is collected from
netstat– While running netperf, information is collected from
vmstat.
![Page 4: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/4.jpg)
Laboratory testing● Hardware:
● Motherboard: Supermicro 370DE6● Memory: 1GB SDRAM● CPU: 1GHz Pentium III● NIC: 3COM 3C985B-SX
● Operating systems:● Linux: 2.4.9● NetBSD: current● Windows 2000: service pack 2
● RTT=0.227ms
![Page 5: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/5.jpg)
Point to point - throughput
4KB 8KB 16KB 32KB 64KB 128KB
256KB
512KB
1024KB
2048KB
0
100
200
300
400
500
600
700
800
900
1000
Throughput
Linux(1500)
Windows(1500)
NetBSD(1500)
Linux(9000)
Windows(9000)
NetBSD(9000)
TCP window
Mbit/s
![Page 6: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/6.jpg)
Point to point - CPU load
4KB 8KB 16KB 32KB 64KB 128KB
256KB
512KB
1024KB
2048KB
0
10
20
30
40
50
60
70
80
90
100
CPU load
Linux(1500)-t
Linux(1500)-r
NetBSD(1500)-tNetBSD(1500)-r
Linux(9000)-t
Linux(9000)-rNetBSD(9000)-t
NetBSD(9000)-r
TCP window
%
![Page 7: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/7.jpg)
Point to point - interrupts/s
4KB 8KB 16KB 32KB 64KB 128KB
256KB
512KB
1024KB
2048KB
0250050007500
10000125001500017500200002250025000275003000032500350003750040000
Interrupts
Linux(1500)-t
Linux(1500)-r
NetBSD(1500)-t
NetBSD(1500)-r
Linux(9000)-t
Linux(9000)-r
NetBSD(9000)-t
NetBSD(9000)-r
TCP window
Int/s
![Page 8: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/8.jpg)
Analyzing results
● Problems with using tcpdump● Negative effect on throughput● Drops many packets● Interrupt Coalescing
● Gives inaccurate time stamps● Low performance using Windows
2000 probably related to driver.
MTU Linux NetBSD Windows 1500 792 640 3759000 988 988 692
![Page 9: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/9.jpg)
Real world experiments
MTU size: n=normal(1500) j=jumboframe(4470)
Naming: nn
country: c=Czech republicf=Finlandn=Norways=Switzerland
These tests have been possiblethanks to the good help from:●Jari Miettinen, CSC●Tuomas Toropainen, CSC●Simon Leinen, SWITCH●Sven Ubik, CESNET
Czech Finland Norway Switzerland NICCzech 43.2 51.1 31.1Allied Telesyn AT2970SX/SCFinland 43.2 20.5 57.43COM 3C996Norway 51.1 20.5 65.53COM 3C985BSwitzerland 31.1 57.4 65.5 SysKonnect SK-9843
RTT:
![Page 10: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/10.jpg)
Finland - Norway
64KB
128KB
256KB
512KB
1MB 2MB 3MB 4MB 5MB 6MB 7MB 8MB 10MB
0
100
200
300
400
500
600
700
800
900
1000
Throughput
nn->fj
fj->nn
nj->fj
fj->nj
TCP window
Mbit/s
![Page 11: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/11.jpg)
Finland - Switzerland
64KB
128KB
256KB
512KB
1MB 2MB 3MB 4MB 5MB 6MB 7MB 8MB 10MB
0255075
100125150175200225250275300325350375400
Throughput
sn->fj
fj->snsj->fjfj->sj
TCP window
Mbit/s
![Page 12: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/12.jpg)
Norway - Switzerland
64KB
128KB
256KB
512KB
1MB 2MB 3MB 4MB 5MB 6MB 7MB 8MB 10MB
0
25
50
75
100
125
150
175
200
225
Throughput
nn->sn
sn->nn
nj->sj
sj->nj
TCP window
Mbit/s
![Page 13: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/13.jpg)
Variations over a day
0 1 2 3 4 5 6 7 8 9 10
11
12
13
14
15
16
17
18
19
20
21
22
23
0
100
200
300
400
500
600
700
800
900
1000
Throughput
nj-fj
nj-sj
Time of day
Mbit/s
![Page 14: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/14.jpg)
Czech-Finland-Norway
64KB
128KB
256KB
512KB
1MB 2MB 3MB 4MB 5MB 6MB 7MB 8MB 10MB
0
50
100
150
200
250
300
350
400
450
500
550
600
Throughput
nn-cncn-nn
fj-cn
cn-fj
TCP window
Mbit/s
![Page 15: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/15.jpg)
Analyzing results
● Good performance using jumboframes.
● Problems with large TCP window sizes● Asymetric results
– Probably driver and hardware related● Variance in results to Switzerland
caused by SACKs and retransmits
Czech Finland Norway SwitzerlandCzech 482 601 -Finland 219 130 48Norway 181 230 79Switzerland - 107 123
Fro
m
To
MTU=1500Czech Finland Norway Switzerland
Czech - - -Finland - 568 364Norway - 953 219Switerland - 396 215
MTU=4470
To
Fro
m
![Page 16: TCP Performance over Gigabit Networks Arne Øslebø NORDUnet2002.](https://reader036.fdocuments.us/reader036/viewer/2022083008/56649ee15503460f94bf24fe/html5/thumbnails/16.jpg)
Conclusions and further work● Conclusions:
– Diffucult to achieve high performance– Still possible if you use:
● A good PC● A good network card● A good driver for the card
– Large TCP windows leads to performance degradation.– Default maximum TCP window size too small for high
speed networks.● Further work:
– Will investigate asymetric results in more details.● Use DAG cards for packet capture to get accurate TCP traces.
● Full report will be posted on http://www.uninett.no/prosjekt/tcp-revisited/