Olivier Martin, CERN NEC’2005 Conference, VARNA (Bulgaria)
description
Transcript of Olivier Martin, CERN NEC’2005 Conference, VARNA (Bulgaria)
-
The ongoing evolution from Packet based networks to Hybrid Networks in Research & Education NetworksOlivier Martin, CERNNEC2005 Conference, VARNA (Bulgaria)
-
Presentation Outline
The demise of conventional packet based networks in the R&E communityThe advent of community managed dark fiber networksThe Grid & its associated Wide Area Networking challengeson-demand Lambda GridsEthernet over SONET & new standardsWAN-PHY, GFP, VCAT/LCAS, G.709, OTN
-
OC-768c40-GE
-
Internet Backbone SpeedsT1 LinesT3 linesOC3cOC12cIP/ATM-VCsMBPS
Chart1
0.056
0.256
4.5
4.5
10
20
40
180
360
900
2000
4000
8000
100000
5000000
Backbone Speed
Internet Backbone Speed (in Mbps)
Sheet1
YearBackbone Speed
19860.05656,000
19870.256256,000
19984.51,560,000
19894.51,560,000
1990103,000,000
1991204,500,000
19924010,000,000
199318045,000,000
1994360130,000,000
1995900300,000,000
19962000900,000,000
199740002,000,000,000
199880004,000,000,000
1999100000100,000,000,000
200050000005,000,000,000,000
Sheet1
Backbone Speed
Internet Backbone Speed (in Mbps)
Sheet2
Sheet3
-
High Speed IP Network Transport TrendsHigher Speed, Lower cost, complexity and overheadB-ISDNATMSONET/SDHIPOpticalMultiplexing, protection and management at every layerSignalling
-
Network ExponentialsNetwork vs. computer performanceComputer speed doubles every 18 monthsNetwork speed doubles every 9 monthsDifference = order of magnitude per 5 years1986 to 2000Computers: x 500Networks: x 340,0002001 to 2010Computers: x 60Networks: x 4000Moores Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.
l
Know the userBW requirements# of usersCABA -> Lightweight users, browsing, mailing, home useB -> Business applications, multicast, streaming, VPNs, mostly LANC -> Special scientific applications, computing, data grids, virtual-presenceADSLGigE LAN(3 of 12)F(t)
l
What the userBW requirementsTotal BWCABA -> Need full Internet routing, one to manyB -> Need VPN services on/and full Internet routing, several to several C -> Need very fat pipes, limited multiple Virtual Organizations, few to fewADSLGigE LAN(4 of 12)
l
So what are the factsCosts of fat pipes (fibers) are one/third of equipment to light them upIs what Lambda salesmen told Cees de Laat (University of Amsterdam & Surfnet) Costs of optical equipment 10% of switching 10 % of full routing equipment for same throughput100 Byte packet @ 10 Gb/s -> 80 ns to look up in 100 Mbyte routing table (light speed from me to you on the back row!)Big sciences need fat pipesBottom line: create a hybrid architecture which serves all users in one coherent and cost effective way(5 of 12)
-
Utilization trends
GbpsNetwork Capacity LimitJan 2005
-
Todays hierarchical IP networkUniversityNational or Pan-National IP NetworkOther national networksNREN ANREN BNREN CNREN D
-
Tomorrows peer to peer IP networkWorldUniversityServerWorldWorldNational DWDM NetworkNREN ANREN BNREN CNREN DChildLightpathsChild Lightpaths
-
Creation of application VPNsCommodityInternetBio-informaticsNetworkUniversityUniversityUniversityCERNUniversityUniversityHigh Energy Physics NetworkeVLBI NetworkDeptResearch NetworkDirect connect bypasses campus firewall
-
Production vs Research Campus NetworksIncreasingly campuses are deploying parallel networks for high end usersReduces costs by providing high end network capability to only those who need itLimitations of campus firewall and border router are eliminatedMany issues in regards to security, back door routing, etcCampus networks may follow same evolution as campus computingDiscipline specific networks being extended into the campus
-
UCLP intended for projects like National LambdaRail
-
GEANT2 POP Design
EMBED Word.Picture.8
_1147001627.doc
Nx10Gbps to other GANT2 PoP
2x10Gbps to local NREN
Dark fibre to other GANT2 PoP
GANT2 PoP
Juniper M-160
DWDM
-
UltraLight Optical Exchange PointL1, L2 and L3 servicesInterfaces1GE and 10GE10GE WAN-PHY (SONET friendly)Hybrid packet- and circuit-switched PoPInterface between packet- & circuit-switched networksControl plane is L3Photonic switchCalient or Glimmerglass Photonic Cross Connect Switch
-
LHC Data Grid HierarchyTier 1Online System CERN 700k SI95 ~1 PB Disk; Tape RobotFNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center InstituteInstituteInstituteInstitute ~0.25TIPSWorkstations~100-400 MBytes/sec2.5/10 Gbps0.11 GbpsPhysicists work on analysis channelsEach institute has ~10 physicists working on one or more channelsPhysics data cache~PByte/sec10 Gbps~2.5 GbpsTier 0 +1Tier 3Tier 4Tier 2ExperimentCERN/Outside Resource Ratio ~1:2 Tier0/( Tier1)/( Tier2) ~1:1:1
-
Deploying the LHC [email protected] LHC Computing Centre
-
What you [email protected]
-
Main Networking Challenges
Fulfill the, yet unproven, assertion that the network can be nearly transparent to the GridDeploy suitable Wide Area Network infrastructure (50-100 Gb/s)Deploy suitable Local Area Network infrastructure (matching or exceeding that of the WAN)Seamless interconnection of LAN & WAN infrastructuresfirewall?End to End issues (transport protocols, PCs (Itanium, Xeon), 10GigE NICs (Intel, S2io), where are we today:memory to memory: 7.5Gb/s (PCI bus limit)memory to disk: 1.2MB (Windows 2003 server/NewiSys)disk to disk: 400MB (Linux), 600MB (Windows)
-
Main TCP issuesDoes not scale to some environmentsHigh speed, high latencyNoisyUnfair behaviour with respect to:Round Trip Time (RTTFrame size (MSS)Access BandwidthWidespread use of multiple streams in order to compensate for inherent TCP/IP limitations (e.g. Gridftp, BBftp):Bandage rather than a cureNew TCP/IP proposals in order to restore performance in single stream environmentsNot clear if/when it will have a real impactIn the mean time there is an absolute requirement for backbones with:Zero packet losses,And no packet re-orderingWhich re-inforces the case for lambda Grids
-
TCP dynamics(10Gbps, 100ms RTT, 1500Bytes packets)Window size (W) = Bandwidth*Round Trip TimeWbits = 10Gbps*100ms = 1GbWpackets = 1Gb/(8*1500) = 83333 packetsStandard Additive Increase Multiplicative Decrease (AIMD) mechanisms:W=W/2 (halving the congestion window on loss event)W=W + 1 (increasing congestion window by one packet every RTT)Time to recover from W/2 to W (congestion avoidance) at 1 packet per RTT:RTT*Wp/2 = 1.157 hourIn practice, 1 packet per 2 RTT because of delayed acks, i.e. 2.31 hourPackets per second:RTT*Wpackets = 833333 packets
-
Single TCP stream performance under periodic lossesTCP throughput much more sensitive to packet loss in WANs than LANsTCPs congestion control algorithm (AIMD) is not well-suited to gigabit networksThe effect of packets loss can be disastrousTCP is inefficient in high bandwidth*delay networksThe future performance-outlook for computational grids looks bad if we continue to rely solely on the widely-deployed TCP RENOLoss rate =0.01%:LAN BW utilization= 99%WAN BW utilization=1.2%Bandwidth available = 1 Gbps
Chart7
1.265957446822.8723404255
4.031914893635.4255319149
12.765957446864.6808510638
19.893617021377.7659574468
27.872340425597.4468085106
34.468085106497.9787234043
39.468085106498.4042553191
53.617021276698.5106382979
0.00000498.9361702128
0.000003333399.5744680851
0.00000299.6808510638
0.000001666799.7872340426
0.0000013333100
WAN (RTT=120ms)
LAN (RTT=0.04 ms)
Packet Loss frequency (%)
Bandwidth Utilization (%)
Effect of packet loss
CHI
1/PBW (MbpsBW (Mbpsbw utilsation50900150200250300350400450500550600650
0
10100.26781249780.37555315790.0023360.0001297778215
10010.8468974791.18760336130.023360.0012977778333
10000.12.67812497833.75553157870.23360.0129777778608
100000.018.468974789911.876033613411.91.26595744682.3360.1297777778731
1000000.00126.781249782737.555315787237.94.031914893623.361.2977777778
10000000.000184.6897478992118.760336134512012.7659574468233.612.9777777778570776.255707763428082.191780822916
25000000.00004133.9062489134187.77657893618719.893617021358432.4444444444921
50000000.00002189.3720332998265.556184627426227.8723404255116864.8888888889925
75000000.0000133333231.932426569325.238575188732434.4680851064175297.3333333333926
100000000.00001267.8124978268375.553157872137139.46808510642336129.7777777778930
175000000.0000057143354.2826336224496.810129907250453.61702127664088227.1111111111936
250000000.000004423.4487394958593.80168067235840324.4444444444937
300000000.0000033333463.864853138650.47715037747008389.3333333333938
500000000.000002598.8469503648839.762390166711680648.8888888889940
600000000.0000016667656.0039664159919.913608077514016778.6666666667940
750000000.0000013333733.43473120771028.494680544217520973.3333333333940
1000000000.000001846.89747899161187.6033613445233601297.7777777778
10000000000.00000012678.1249782683755.531578720723360012977.7777777778
CHI
00215
00333
00608
00731
00916
00921
00925
00926
00930
00936
00937
00938
00940
00940
00940
00940
00940
00940
MSS*C/ (RTT*sqrt(p))
WAN (RTT=120ms)
LAN (RTT=0.04 ms)
Packet Loss frequency (one packet every x packets transmitted)
Throuhput (Mbps)
Effect of packet loss
Gva
022.8723404255
035.4255319149
064.6808510638
077.7659574468
097.4468085106
097.9787234043
098.4042553191
098.5106382979
098.9361702128
099.5744680851
099.6808510638
099.7872340426
0100
0100
0100
WAN (RTT=120ms)
LAN (RTT=0.04 ms)
Packet Loss frequency (one packet every x packets transmitted)
Bandwidth Utilisation %
Effect of packet loss
Sheet3
00215
00333
00608
00731
00916
00921
00925
00926
00930
00936
00937
00938
00940
00940
00940
940
940
940
MSS*C/ (RTT*sqrt(p))
WAN (RTT=120ms)
LAN (RTT=0.04 ms)
Packet Loss frequency (%)
Throuhput (Mbps)
Effect of packet loss
022.8723404255
035.4255319149
064.6808510638
077.7659574468
097.4468085106
097.9787234043
098.4042553191
098.5106382979
098.9361702128
099.5744680851
099.6808510638
099.7872340426
0100
WAN (RTT=120ms)
LAN (RTT=0.04 ms)
Packet Loss frequency (%)
Bandwidth Utilisation (%)
Effect of packet loss
w05gva -> w06gva
w=1M
RTT=
txqueuelen = 100
Loss rate (%)Rate (Mbit/s)Link Utilisation (%)
10101021522.87234042550.0005432558
205533335.42553191490.0007015015
502260864.68085106380.0009605263
1001173177.76595744680.0015978112
5000.20.291697.44680851060.0063755459
7500.13333333330.133333333392197.97872340430.0095114007
10000.10.192598.40425531910.012627027
50000.020.0292698.51063829790.0630669546
100000.010.0193098.93617021280.1255913978
1000000.0010.00193699.57446808511.2478632479939
10000000.00010.000193799.680851063812.4653148346
25000000.000040.0000493899.787234042631.1300639659
50000000.000020.0000294010062.1276595745
75000000.00001333330.000013333394010093.1914893617
100000000.000010.00001940100124.2553191489
175000000.00000571430.0000057143940100217.4468085106
250000000.0000040.000004940100310.6382978723
300000000.00000333330.0000033333940100372.7659574468
09
-
ResponsivenessLarge MTU accelerates the growth of the windowTime to recover from a packet loss decreases with large MTULarger MTU reduces overhead per frames (saves CPU cycles, reduces the number of packets) r =C . RTT2 . MSS2C : Capacity of the linkTime to recover from a single packet loss:
-
Internet2 land speed record history (IPv4 & IPv6) period 2000-2004
Chart1
MonthMonthMonthMonth
0.760.956Mar-00Mar-00
0.4020.402Apr-02Apr-02
Sep-02Sep-020.4830.483
Oct-02Oct-020.3480.348
0.9230.923Nov-02Nov-02
2.382.38Feb-03Feb-03
May-03May-030.9830.983
5.445.44Oct-03Oct-03
5.645.64Nov-03Nov-03
Nov-03Nov-0344
Feb-046.25Feb-04Feb-04
4.2226Apr-04Apr-04Apr-04
May-047.09May-04May-04
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
DataTAG-Web-Hits
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
Hits/month
Month
Hits
DataTAG Web site
DataTAG-Web-Hits-v2
Month
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
31379
25998
23407
26423
28340
22866
27250
40518
29530
28480
30335
58530
79073
DataTAG
Month
Hits
IPv4-IPv6-Gbs
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) multiple streams
Month
Internet2 landspeed record history(in Gigabit/second)
IPv4-IPv6-Tbms
MonthMonth
4278Mar-00
4933Apr-02
Sep-021215
Oct-025154
10136Nov-02
23886Feb-03
May-036947
38420Oct-03
61752Nov-03
Nov-0346156
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
Month
Internet2 landspeed record history(in terabit-meters/second)
IP-v4-IPv6-Gbs-v1
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) multiple streams
Month
Evolution of the I2LSR in Gigabit/second
IPv4-IPv6-Tbms-v1
MonthMonth
4278Mar-00
4933Apr-02
Sep-021215
Oct-025154
10136Nov-02
23886Feb-03
May-036947
38420Oct-03
61752Nov-03
Nov-0346156
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
Month
Evolution of the I2LSR using terabit-meters/second metrics
IPv4-v6-Tbms
MonthMonthMonthMonthMonthMonth
Mar-00Mar-00Mar-00Mar-004278Mar-00
Apr-02Apr-02Apr-02Apr-024933Apr-02
Sep-02Sep-02Sep-02Sep-02Sep-021215
Oct-02Oct-02Oct-02Oct-02Oct-025154
Nov-02Nov-02Nov-02Nov-0210136Nov-02
Feb-03Feb-03Feb-03Feb-0323886Feb-03
May-03May-03May-03May-03May-036947
Oct-03Oct-03Oct-03Oct-0338420Oct-03
Nov-03Nov-03Nov-03Nov-0361752Nov-03
Nov-03Nov-03Nov-03Nov-03Nov-0346156
Feb-04Feb-04Feb-04Feb-0468431Feb-04
Apr-04Apr-04Apr-04Apr-0469073Apr-04
May-04May-04May-04May-0477699May-04
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
MONTH
TERABIT-METERS/SECOND
EVOLUTION OF THE I2LSR IN TERABITS-METERS/SECOND
IPv4-v6-Gbs
MonthMonthMonthMonth
0.760.956Mar-00Mar-00
0.4020.402Apr-02Apr-02
Sep-02Sep-020.4830.483
Oct-02Oct-020.3480.348
0.9230.923Nov-02Nov-02
2.382.38Feb-03Feb-03
May-03May-030.9830.983
5.445.44Oct-03Oct-03
5.645.64Nov-03Nov-03
Nov-03Nov-0344
Feb-046.25Feb-04Feb-04
4.2226Apr-04Apr-04Apr-04
May-047.09May-04May-04
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
MONTH
Gigabits/second
Category
EVOLUTION OF THE I2LSR IN GIGABITS/SECOND
IPv4-IPv6
IPv4 (Gb/s) single streamIPv4 (Gb/s) multiple streamsIPv6 (Gb/s) single streamIPv6 (Gb/s) multiple streamsIPv4 terabit-meters/second)IPv6 (terabit-meters/second)Owner(s)
Month
Mar-000.7600.9564,278Microsoft, Qwest, University of Washington, USC ISI
Apr-020.4020.4024,933University of Alberta, University of Amsterdam (UvA), Surfnet
Sep-020.4830.4831,215ARNES, DANTE, RedIris
Oct-020.3480.3485,154ARNES, DANTE, RedIris
Nov-020.9230.92310,136Caltech, DataTAG, Nikhef, SLAC, UvA
Feb-032.382.3823,886Caltech, CERN, DataTAG, Los Alamos National Laboratory (LANL), SLAC
May-030.9830.9836,947Caltech, CERN, DataTAG
Oct-035.445.4438,420Caltech, CERN, DataTAG
Nov-035.645.6461,752Caltech, CERN, DataTAG
Nov-034446,156Caltech, CERN, DataTAG
Feb-046.2568,431Caltech, CERN, DataTAG (multi-stream)
Apr-044.222669,073SUNET
May-047.0977,699Caltech, CERN, DataTAG (multi-stream)
IPv4-IPv6
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
Datag related Web sites
MonthMonthMonth
4241Apr-02Apr-02
4910May-02May-02
3889Jun-02Jun-02
6949Jul-02Jul-02
8788Aug-02Aug-02
17356Sep-02Sep-02
18616Oct-02Oct-02
20420Nov-02Nov-02
18851Dec-02Dec-02
37266Jan-03Jan-03
36076Feb-03Feb-03
31379Mar-03Mar-03
25998Apr-03Apr-03
23407May-03May-03
26423Jun-03Jun-03
28340Jul-03Jul-03
22866Aug-03Aug-03
27250Sep-03Sep-03
405181058Oct-03
29530297Nov-03
2848037916569
303353351968
58530306915
DataTAG
Telecom
ICT4D
Month
Hits/month
DataTAG related Web sites
DataTAG Web site activity
Month
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
31379
25998
23407
26423
28340
22866
27250
40518
29530
28480
30335
58530
79073
DataTAG
Month
Hits/month
DataTAG Web site
DataTAG Tables
DataTAGGridcafeTelecomICT4D
Month
Apr-024241
May-024910
Jun-023889
Jul-026949
Aug-028788
Sep-0217356
Oct-0218616
Nov-0220420
Dec-0218851
Jan-0337266
Feb-0336076
Mar-0331379
Apr-0325998
May-0323407
Jun-0326423
Jul-0328340
Aug-0322866
Sep-0327250
Oct-03405181716761058
Nov-0329530379547297
Dec-032848031876437916569
Jan-04303352700003351968
Feb-0458530291255306915
Mar-04790732699846001339
Apr-0446165497284332818
May-04409906141719-May
Jun-04
Sheet1
year16002000250030000.800.710.630.5
19984.05.06.08.0400.00400.00400.00400.00
199920.025.031.038.080.0080.0080.0080.00
200044.455.669.483.336.0036.0036.0036.00
2001160.0200.0250.0300.010.0010.0010.0010.00Based on actual 155Mbps pri
20021012.71265.81582.31898.71.581.581.581.58Based on actual 622Mbps pri
20031265.81582.31977.82373.41.261.121.000.79
20041582.31977.82472.32966.81.010.800.630.40
20051977.82472.33090.43708.50.810.570.400.20
20062472.33090.43863.04635.60.650.400.250.10
20073090.43863.04828.75794.50.520.290.160.05
20083863.04828.76035.97243.10.410.200.100.02
year1600.02000.02500.03000.0
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20031426.31782.82228.62674.3
20042008.82511.13138.83766.6
20052829.43536.74420.95305.0
20063985.04981.36226.67471.9
20075612.77015.98769.810523.8
20087905.29881.512351.914822.2
year1600.02000.02500.03000.0
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20031607.42009.22511.63013.9
20042551.43189.33986.64783.9
20054049.95062.36327.97593.5
20066428.48035.510044.312053.2
200710203.812754.715943.419132.0
200816196.420245.625306.930368.3
year1600200025003000
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20032025.32531.63164.63797.5
20044050.65063.36329.17594.9
20058101.310126.612658.215189.9
200616202.520253.225316.530379.7
200732405.140506.350632.960759.5
200864810.181012.7101265.8121519.0
622 Kb[s-0.20-0.29-0.37-0.50Hypothesis 622Mbps=2*155Mbps
20013100.003100.003100.003100.00
2002982.76982.76982.76982.76
2003786.21697.76619.14491.38
2004628.97495.41390.06245.69
2005503.17351.74245.74122.85
2006402.54249.74154.8161.42
2007322.03177.3197.5330.71
2.5 Gb[s-0.20-0.29-0.33-0.50Hypothesis 2.5Gbps=2*622Mbps
20016200.006200.006200.006200.00
20021965.521965.521965.521965.52
20031572.421395.521238.28982.76
20041257.93990.82780.11491.38
20051006.35703.48491.47245.69
2006805.08499.47309.63122.85
2007644.06354.62195.0761.42
1 Gb[s-0.20-0.29-0.33-0.50EuropeUSA
200110000.0010000.0010000.0010000.00
20021580.001580.001580.001580.00
20031264.001121.80995.40790.00
20041011.20796.48627.10395.00376.26156.78206.9414.4966992465
2005808.96565.50395.07197.50237.0498.77130.3753.6914786909
2006647.17401.50248.9098.75149.3462.2282.1466.9621616327
2007517.73285.07156.8049.3894.0839.2051.7585.0313163589
2008414.19202.4098.7924.6959.2724.7032.60104.2952653753
Sheet2
0.800.710.630.5
400.00400.00400.00400.001998
80.0080.0080.0080.001999
36.0036.0036.0036.002000
10.0010.0010.0010.002001
8.007.106.305.002002
6.405.043.972.502003
5.123.582.501.252004
4.102.541.580.632005
3.281.800.990.312006
2.621.280.630.162007
22522522522520012250
36340846058020022900
578734932148020033700
87912571800360020044500
DataTAG-Web-Hits
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
Hits/month
Month
Hits
DataTAG Web site
DataTAG-Web-Hits-v2
Month
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
31379
25998
23407
26423
28340
22866
27250
40518
29530
28480
30335
58530
79073
DataTAG
Month
Hits
IPv4-IPv6-Gbs
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) multiple streams
Month
Internet2 landspeed record history(in Gigabit/second)
IPv4-IPv6-Tbms
MonthMonth
4278Mar-00
4933Apr-02
Sep-021215
Oct-025154
10136Nov-02
23886Feb-03
May-036947
38420Oct-03
61752Nov-03
Nov-0346156
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
Month
Internet2 landspeed record history(in terabit-meters/second)
IP-v4-IPv6-Gbs-v1
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) multiple streams
Month
Evolution of the I2LSR in Gigabit/second
IPv4-IPv6-Tbms-v1
MonthMonth
4278Mar-00
4933Apr-02
Sep-021215
Oct-025154
10136Nov-02
23886Feb-03
May-036947
38420Oct-03
61752Nov-03
Nov-0346156
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
Month
Evolution of the I2LSR using terabit-meters/second metrics
IPv4-v6-Tbms
MonthMonthMonthMonthMonthMonth
Mar-00Mar-00Mar-00Mar-004278Mar-00
Apr-02Apr-02Apr-02Apr-024933Apr-02
Sep-02Sep-02Sep-02Sep-02Sep-021215
Oct-02Oct-02Oct-02Oct-02Oct-025154
Nov-02Nov-02Nov-02Nov-0210136Nov-02
Feb-03Feb-03Feb-03Feb-0323886Feb-03
May-03May-03May-03May-03May-036947
Oct-03Oct-03Oct-03Oct-0338420Oct-03
Nov-03Nov-03Nov-03Nov-0361752Nov-03
Nov-03Nov-03Nov-03Nov-03Nov-0346156
Feb-04Feb-04Feb-04Feb-0468431Feb-04
Apr-04Apr-04Apr-04Apr-0469073Apr-04
May-04May-04May-04May-0477699May-04
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
IPv4 terabit-meters/second)
IPv6 (terabit-meters/second)
MONTH
TERABIT-METERS/SECOND
EVOLUTION OF THE I2LSR IN TERABITS-METERS/SECOND
IPv4-v6-Gbs
MonthMonthMonthMonth
0.760.956Mar-00Mar-00
0.4020.402Apr-02Apr-02
Sep-02Sep-020.4830.483
Oct-02Oct-020.3480.348
0.9230.923Nov-02Nov-02
2.382.38Feb-03Feb-03
May-03May-030.9830.983
5.445.44Oct-03Oct-03
5.645.64Nov-03Nov-03
Nov-03Nov-0344
Feb-046.25Feb-04Feb-04
4.2226Apr-04Apr-04Apr-04
May-047.09May-04May-04
IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams
IPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streams
MONTH
Gigabits/second
Category
EVOLUTION OF THE I2LSR IN GIGABITS/SECOND
Chart3
IPv4 (Gb/s) single stream0.760.402IPv4 (Gb/s) single streamIPv4 (Gb/s) single stream0.9232.38IPv4 (Gb/s) single stream5.445.64IPv4 (Gb/s) single streamIPv4 (Gb/s) single stream4.2226IPv4 (Gb/s) single stream
IPv4 (Gb/s) multiple streams0.9560.402IPv4 (Gb/s) multiple streamsIPv4 (Gb/s) multiple streams0.9232.38IPv4 (Gb/s) multiple streams5.445.64IPv4 (Gb/s) multiple streams6.25IPv4 (Gb/s) multiple streams7.09
IPv6 (Gb/s) single streamIPv6 (Gb/s) single streamIPv6 (Gb/s) single stream0.4830.348IPv6 (Gb/s) single streamIPv6 (Gb/s) single stream0.983IPv6 (Gb/s) single streamIPv6 (Gb/s) single stream4IPv6 (Gb/s) single streamIPv6 (Gb/s) single streamIPv6 (Gb/s) single stream
IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams0.4830.348IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams0.983IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams4IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams
Month
Mar-00
Apr-02
Sep-02
Oct-02
Nov-02
Feb-03
May-03
Oct-03
Nov-03
Nov-03
Feb-04
Apr-04
May-04
Type
Gigabit/second
Month
Evolution of Internet2 Landspeed record
IPv4-IPv6
IPv4 (Gb/s) single streamIPv4 (Gb/s) multiple streamsIPv6 (Gb/s) single streamIPv6 (Gb/s) multiple streamsIPv4 terabit-meters/second)IPv6 (terabit-meters/second)Owner(s)
Month
Mar-000.7600.9564,278Microsoft, Qwest, University of Washington, USC ISI
Apr-020.4020.4024,933University of Alberta, University of Amsterdam (UvA), Surfnet
Sep-020.4830.4831,215ARNES, DANTE, RedIris
Oct-020.3480.3485,154ARNES, DANTE, RedIris
Nov-020.9230.92310,136Caltech, DataTAG, Nikhef, SLAC, UvA
Feb-032.382.3823,886Caltech, CERN, DataTAG, Los Alamos National Laboratory (LANL), SLAC
May-030.9830.9836,947Caltech, CERN, DataTAG
Oct-035.445.4438,420Caltech, CERN, DataTAG
Nov-035.645.6461,752Caltech, CERN, DataTAG
Nov-034446,156Caltech, CERN, DataTAG
Feb-046.2568,431Caltech, CERN, DataTAG (multi-stream)
Apr-044.222669,073SUNET
May-047.0977,699Caltech, CERN, DataTAG (multi-stream)
Datag related Web sites
MonthMonthMonth
4241Apr-02Apr-02
4910May-02May-02
3889Jun-02Jun-02
6949Jul-02Jul-02
8788Aug-02Aug-02
17356Sep-02Sep-02
18616Oct-02Oct-02
20420Nov-02Nov-02
18851Dec-02Dec-02
37266Jan-03Jan-03
36076Feb-03Feb-03
31379Mar-03Mar-03
25998Apr-03Apr-03
23407May-03May-03
26423Jun-03Jun-03
28340Jul-03Jul-03
22866Aug-03Aug-03
27250Sep-03Sep-03
405181058Oct-03
29530297Nov-03
2848037916569
303353351968
58530306915
DataTAG
Telecom
ICT4D
Month
Hits/month
DataTAG related Web sites
DataTAG Web site activity
Month
4241
4910
3889
6949
8788
17356
18616
20420
18851
37266
36076
31379
25998
23407
26423
28340
22866
27250
40518
29530
28480
30335
58530
79073
DataTAG
Month
Hits/month
DataTAG Web site
DataTAG Tables
DataTAGGridcafeTelecomICT4D
Month
Apr-024241
May-024910
Jun-023889
Jul-026949
Aug-028788
Sep-0217356
Oct-0218616
Nov-0220420
Dec-0218851
Jan-0337266
Feb-0336076
Mar-0331379
Apr-0325998
May-0323407
Jun-0326423
Jul-0328340
Aug-0322866
Sep-0327250
Oct-03405181716761058
Nov-0329530379547297
Dec-032848031876437916569
Jan-04303352700003351968
Feb-0458530291255306915
Mar-04790732699846001339
Apr-0446165497284332818
May-04409906141719-May
Jun-04
Sheet1
year16002000250030000.800.710.630.5
19984.05.06.08.0400.00400.00400.00400.00
199920.025.031.038.080.0080.0080.0080.00
200044.455.669.483.336.0036.0036.0036.00
2001160.0200.0250.0300.010.0010.0010.0010.00Based on actual 155Mbps pri
20021012.71265.81582.31898.71.581.581.581.58Based on actual 622Mbps pri
20031265.81582.31977.82373.41.261.121.000.79
20041582.31977.82472.32966.81.010.800.630.40
20051977.82472.33090.43708.50.810.570.400.20
20062472.33090.43863.04635.60.650.400.250.10
20073090.43863.04828.75794.50.520.290.160.05
20083863.04828.76035.97243.10.410.200.100.02
year1600.02000.02500.03000.0
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20031426.31782.82228.62674.3
20042008.82511.13138.83766.6
20052829.43536.74420.95305.0
20063985.04981.36226.67471.9
20075612.77015.98769.810523.8
20087905.29881.512351.914822.2
year1600.02000.02500.03000.0
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20031607.42009.22511.63013.9
20042551.43189.33986.64783.9
20054049.95062.36327.97593.5
20066428.48035.510044.312053.2
200710203.812754.715943.419132.0
200816196.420245.625306.930368.3
year1600200025003000
19984.05.06.08.0
199920.025.031.038.0
200044.455.669.483.3
2001160.0200.0250.0300.0
20021012.71265.81582.31898.7
20032025.32531.63164.63797.5
20044050.65063.36329.17594.9
20058101.310126.612658.215189.9
200616202.520253.225316.530379.7
200732405.140506.350632.960759.5
200864810.181012.7101265.8121519.0
622 Kb[s-0.20-0.29-0.37-0.50Hypothesis 622Mbps=2*155Mbps
20013100.003100.003100.003100.00
2002982.76982.76982.76982.76
2003786.21697.76619.14491.38
2004628.97495.41390.06245.69
2005503.17351.74245.74122.85
2006402.54249.74154.8161.42
2007322.03177.3197.5330.71
2.5 Gb[s-0.20-0.29-0.33-0.50Hypothesis 2.5Gbps=2*622Mbps
20016200.006200.006200.006200.00
20021965.521965.521965.521965.52
20031572.421395.521238.28982.76
20041257.93990.82780.11491.38
20051006.35703.48491.47245.69
2006805.08499.47309.63122.85
2007644.06354.62195.0761.42
1 Gb[s-0.20-0.29-0.33-0.50EuropeUSA
200110000.0010000.0010000.0010000.00
20021580.001580.001580.001580.00
20031264.001121.80995.40790.00
20041011.20796.48627.10395.00376.26156.78206.9414.4966992465
2005808.96565.50395.07197.50237.0498.77130.3753.6914786909
2006647.17401.50248.9098.75149.3462.2282.1466.9621616327
2007517.73285.07156.8049.3894.0839.2051.7585.0313163589
2008414.19202.4098.7924.6959.2724.7032.60104.2952653753
Sheet2
0.800.710.630.5
400.00400.00400.00400.001998
80.0080.0080.0080.001999
36.0036.0036.0036.002000
10.0010.0010.0010.002001
8.007.106.305.002002
6.405.043.972.502003
5.123.582.501.252004
4.102.541.580.632005
3.281.800.990.312006
2.621.280.630.162007
22522522522520012250
36340846058020022900
578734932148020033700
87912571800360020044500
-
Layer1/2/3 networking (1) Conventional layer 3 technology is no longer fashionable because of:High associated costs, e.g. 200/300 KUSD for a 10G router interfaces Implied use of shared backbonesThe use of layer 1 or layer 2 technology is very attractive because it helps to solve a number of problems, e.g. 1500 bytes Ethernet frame size (layer1)Protocol transparency (layer1 & layer2)Minimum functionality hence, in theory, much lower costs (layer1&2)
-
Layer1/2/3 networking (2) 0n-demand Lambda Grids are becoming very popular: Pros: circuit oriented model like the telephone network, hence no need for complex transport protocols Lower equipment costs (i.e. in theory a factor 2 or 3 per layer)the concept of a dedicated end to end light path is very elegant Cons: End to end still very loosely defined, i.e. site to site, cluster to cluster or really host to hostHigher circuit costs, Scalability, Additional middleware to deal with circuit set up/tear down, etcExtending dynamic VLAN functionality is a potential nightmare!
-
Lambda Grids What does it mean?
Clearly different things to different people, hence the apparently easy consensus!Conservatively, on demand site to site connectivityWhere is the innovation?What does it solve in terms of transport protocols?Where are the savings?Less interfaces needed (customer) but more standby/idle circuits needed (provider)Economics from the service provider vs the customer perspective?Traditionally, switched services have been very expensive, Usage vs flat chargeBreak even, switches vs leased, few hours/dayWhy would this change? In case there are no savings, why bother?More advanced, cluster to clusterImplies even more active circuits in paralleIs it realistic?Even more advanced, Host to Host or even per flowAll opticalIs it really realisitic?
-
Some ChallengesReal bandwidth estimates given the chaotic nature of the requirements.End-end performance given the whole chain involved (disk-bus-memory-bus-network-bus-memory-bus-disk)Provisioning over complex network infrastructures (GEANT, NRENs etc)Cost model for options (packet+SLAs, circuit switched etc)Consistent Performance (dealing with firewalls)Merging leading edge research with production networking
-
Tentative conclusions
There is a very clear trend towards community managed dark fiber networksAs a consequence National Research & Education Networks are evolving into Telecom Operators, is it right?In the short term, almost certainly YESIn the longer term, probably NOIn many countries, there is NO other way to have affordable access to multi-Gbit/s networks, therefore this is clearly the right moveThe Grid & its associated Wide Area Networking challengeson-demand Lambda Grids are, according to me, extremely doubtful!Ethernet over SONET & new standards will revolutionize the InternetWAN-PHY (IEEE) has, according to me NO future!However, GFP, VCAT/LCAS, G.709, OTN are very likely to have a very bright future.
-
Single TCP stream between Caltech and CERNAvailable (PCI-X) Bandwidth=8.5 Gbps RTT=250ms (16000 km) 9000 Byte MTU15 min to increase throughput from 3 to 6 Gbps
Sending station: Tyan S2882 motherboard, 2x Opteron 2.4 GHz , 2 GB DDR.Receiving station: CERN OpenLab:HP rx4640, 4x 1.5GHz Itanium-2, zx1 chipset, 8GB memoryNetwork adapter: S2IO 10 GbE
Burst of packet lossesSingle packet lossCPU load = 100%
-
High Throughput Disk to Disk Transfers: From 0.1 to 1GByte/sec Server Hardware (Rather than Network) Bottlenecks:Write/read and transmit tasks share the same limited resources: CPU, PCI-X bus, memory, IO chipset PCI-X bus bandwidth: 8.5 Gbps [133MHz x 64 bit]Link aggregation (802.3ad): Logical interface with two physical interfaces on two independent PCI-X buses.LAN test: 11.1 Gbps (memory to memory)
Performance in this range (from 100 MByte/sec up to 1 GByte/sec) is required to build a responsive Grid-based Processing and Analysis System for LHC
- Transferring a TB from Caltech to CERN in 64-bit MS Windows Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec) - 8 TCP streams from CERN to Caltech; 1TB file 3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disksLocal Disk IO 9.6 Gbits/sec (1.2 GBytes/sec read/write, with
-
UltraLight: Developing Advanced Network Services for Data Intensive HEP ApplicationsUltraLight: a next-generation hybrid packet- and circuit-switched network infrastructurePacket switched: cost effective solution; requires ultrascale protocols to share 10G efficiently and fairlyCircuit-switched: Scheduled or sudden overflow demands handled by provisioning additional wavelengths; Use path diversity, e.g. across the US, Atlantic, Canada,Extend and augment existing grid computing infrastructures (currently focused on CPU/storage) to include the network as an integral componentUsing MonALISA to monitor and manage global systemsPartners: Caltech, UF, FIU, UMich, SLAC, FNAL, MIT/Haystack; CERN, NLR, CENIC, Internet2; Translight, UKLight, Netherlight; UvA, UCL, KEK, TaiwanStrong support from Cisco
-
UltraLight MPLS NetworkCompute path from one given node to another such that the path does not violate any constraints (bandwidth/administrative requirements)Ability to set the path the traffic will take through the network (with simple configuration, management, and provisioning mechanisms)Take advantage of the multiplicity of waves/L2 channels across the US (NLR, HOPI, Ultranet and Abilene/ESnet MPLS services) EoMPLS will be used to build layer2 pathsnatural step toward the deployment of GMPLS
-
SummaryFor many years the Wide Area Network has been the bottleneck; this is no longer the case in many countries thus making deployment of a data intensive Grid infrastructure possible!Recent I2LSR records show for the first time ever that the network can be truly transparent and that throughputs are limited by the end hostsChallenge shifted from getting adequate bandwidth to deploying adequate infrastructure to make effective use of it!Some transport protocol issues still need to be resolved; however there are many encouraging signs that practical solutions may now be in sight.1GByte/sec disk to disk challenge. Today: 1 TB at 536 MB/sec from CERN to Caltech Still in Early Stages; Expect Substantial ImprovementsNext generation network and Grid system: UltraLightDeliver the critical missing component for future eScience: the integrated, managed network Extend and augment existing grid computing infrastructures (currently focused on CPU/storage) to include the network as an integral component.
-
10G DataTAG testbed extension to Telecom World 2003 and Abilene/CenicOn September 15, 2003, the DataTAG project was the first transatlantic testbed offering direct 10GigE access using Junipers VPN layer2/10GigE emulation.
And some statistics about the capacity of networks with Ethernet and the Internet backbone compared.The green line shows specially the increase in available bandwidth with DWDM.It shows at the same time that xxGE slower in time will follow backbone bandwidth.
Both statistics show clearly that with the old Ethernet technology building frames in the OS is not anymore possible at this level and something has to be done to offload processor or hub internals with clever hardware oriented algorithms. Stabilizing and improving HW/SW drivers for production useE.g. Reducing extra memory copies; interrupt timing moderation