Post on 16-Aug-2021
1
UCB External ConnectivityUCB External Connectivity
ken lindahlcommunication and network services
uc berkeleylindahl@berkeley.edu
EE122, 7 May 2004
EE122, 7 May 2004 2
CalRENCalREN--2: 1998 to 20032: 1998 to 2003• conceived as a state-wide high-performance, advanced
services network serving California’s research and higher education communities.
• charter members: UC, CSU, Stanford, CalTech, USC
• CalREN-2 network was supposed to OC-48 (2.5Gbps), but what got delivered was two OC-12 (622 Mbps) SONET rings– should have asked for “OC-12c.” oops.
• implemented as one POS (Packet Over SONET) ring and one ATM ring
EE122, 7 May 2004 3
CalRENCalREN--2 northern 2 northern gigaPOPgigaPOP, ca. 11/98, ca. 11/98• “distributed gigaPOP,” with backbone equipment
(SONET MUXes, ATM switches, IP routers) located in campus facilities.
• commercial SONET service for connections between campuses.
EE122, 7 May 2004 4
CalRENCalREN--2 southern 2 southern gigaPOPgigaPOP, ca. 11/98, ca. 11/98
EE122, 7 May 2004 5
CalRENCalREN--2 backbone, circa 11/982 backbone, circa 11/98
EE122, 7 May 2004 6
CalRENCalREN--2: 1998 to 20032: 1998 to 2003• UCB initially connected to CalREN-2 with a single
OC-12 POS link
• subsequently added one, then two, OC-3 (155 Mbps) ATM links for ISP traffic– separate path required in order to control campus ISP costs– diverse routing provided a measure of redundancy, for protection
against failures (but Evans Hall Machine Room was still single point of failure)
2
EE122, 7 May 2004 7
UCB external connectivity, ca. 5/2003 (1)UCB external connectivity, ca. 5/2003 (1)
inr-205
1000
1000
10001000
inr-220inr-230 inr-240 inr-260 inr-270
inr-281
inr-280
inr-120 inr-130 inr-140 inr-160 inr-170 inr-100
100
inr-150
100 100 100 100 100 100
inr-101 inr-102
inr-299
ResHall
EECS Network100
155
1000 1000
Internet2
622
CalREN-2
inr-000
10001000
1000
100
155
CENIC ISP(Level3, Qwest)
100
100
inr-667inr-666
100
100
inr-668
Cogent(SETI@home ISP)
PAIX
100
AffiliatesAffiliates
Affiliates
inr-201 inr-202
100
rate limitingdevices
inr-203
100
Tier 1Backbone
Tier 2Backbone
Seti@Home
BorderRouters
EE122, 7 May 2004 8
shortcomings of the original CalRENshortcomings of the original CalREN--2 network2 network• campus facilities often not hardened against power
outages, flooding, rats, … (read: Evans Hall)
• very little redundancy in network; failures tended to have wide-spread, significant effects
• campus support staff often not funded for 24x7 support
• commercial SONET circuits very expensive, especially after initial 3-year contract expired
• mixing research and ISP traffic on a single backbone had an unexpected result: reliability and availability (for ISP) became more important than flexibility (for research), it became very difficult to deploy new features on the backbone.
EE122, 7 May 2004 9
CENIC next generation backbone considerations (1)CENIC next generation backbone considerations (1)• running out of bandwidth on OC-12 backbone ciruits
• eliminate dependence on flakey campus facilities
• provide redundant connections to campuses
• need to separate production and research networks
• need for true separation of ISP traffic for rate-limiting
capacity planning based on large amounts of headroom needed for occasional use
capacity planning based on expected long-term utilization
outages are undesirable, but acceptable
very low tolerance for outages
run new code as soon as it becomes available
run well-tested code, let somebody else find the bugs
constantly changingavoid change whenever possible
Advanced Services NetworksProduction Networks
EE122, 7 May 2004 10
CENIC next generation backbone considerations (2)CENIC next generation backbone considerations (2)• over-investment in fiber plants causing many carriers to
sell or lease fiber at low cost– vs. recurring high costs for commercial circuits
• development of wave division multiplexing technology– providing multiple high-speed (2.5 Gbps, 10 Gbps, 40 Gbps, …)
channels over a single fiber pair (single fiber strand in some cases)
• researchers requesting “private pipes” between campuses– experiment with non-IP (unroutable) packets– experiment with routing protocols– experiment with “dangerous” packets
• small number of researchers requesting dark fiber between campuses
EE122, 7 May 2004 11
the pyramid!the pyramid!
• plus, ISP “sub-layer” within CalREN-DCEE122, 7 May 2004 12
CalREN Optical Backbone, ca. 3/2004CalREN Optical Backbone, ca. 3/2004
3
EE122, 7 May 2004 13
Wave Division Multiplexing (sort of) demystifiedWave Division Multiplexing (sort of) demystified• multiple channels of data carried over single fiber pair
between WDM terminals
• each channel has its own wavelength (“lambda” or “wave”)
• optical multiplexers combine multiple wavelengths into single composite signal; optical demultiplexers split composite signal into individual wavelengths
• amplification and regeneration needed for longer distances
EE122, 7 May 2004 14
CalREN backbone wavesCalREN backbone waves
San Diego
Tustin
Los Angeles
Santa Barbara
SunnyvaleOakland
San LuisObispo
Soledad
HPRTeragridHPR & DC
CalREN DC
Fergus
Fresno
Bakersfield
Sacramento
UC Davis
Triangle Court
EE122, 7 May 2004 15
Bay Area Metro RingBay Area Metro Ring
EE122, 7 May 2004 16
CalRENCalREN--DC: 10/2003 to presentDC: 10/2003 to present• OC-48c (2.5 Gbps) backbone over private DWDM fiber
• Six GE links to the commodity Internet (Qwest, Level3, Cogent)
• “no-cost” private peerings at PAIX, LAAP, SDNAP
• Serving:– California State University (all campuses)– California Community Colleges (all campuses)– University of California (all campuses)– ~80% of K-12 schools in California
• Estimated user base is 10 million faculty, staff and students!
AOL: 26 million usersCalREN-DC: 10 million usersMSN: 9 million usersEarthlink: 5 million usersNetzero/Juno: 5 million usersComcast/ATT: 3.6 million usersSBC Internet Services: 2.2 million users
EE122, 7 May 2004 17
CalRENCalREN--HPR: 2/2004 to presentHPR: 2/2004 to present• 10Gigabit Ethernet backbone over private DWDM fiber
• Cisco 12400 routers
• 10GE connection to Abilene
• connection to CUDI (Mexico’s version of Internet2)
• Jumbo frames (9180) and IPV6 enabled
• Serving:– University of California (all campuses)– Stanford– University of Southern California – CalTech & Jet Propulsion Laboratory (JPL)
• (future) 10GE to campuses, 12-18 months from now
• (future) PacificWave: distributed exchange point with POPs between LA and Seattle, ideal for landing trans-Pacific fiber links
EE122, 7 May 2004 18
UCB connectivity to CalREN (1)UCB connectivity to CalREN (1)• Total of six GE links:
– GE to CalREN-HPR in CEV near Recreational Sports Facility**– GE to CalREN-DC in CEV near Recreational Sports Facility– GE to CalREN-ISP in CEV near Recreational Sports Facility– GE to CalREN-HPR in Evans Hall**– GE to CalREN-DC in Evans Hall*– GE to CalREN-ISP in Evans Hall*
* share a single 1 Gbps channel over SBC EGM** not yet in service, expect service June 2004
• Redundancy: two campus locations, connected to different CalREN POPs via diverse fiber paths– OAK POP (Emeryville) via dark fiber leased from MFN– SFO POP via SBC Extended GigaMan (EGM) service
4
EE122, 7 May 2004 19
new UCB border routersnew UCB border routers• existing border routers were not up to the task
– inr-666, a cisco 7507, can handle GE interfaces but with only 300-400 Mbps onto the backplane.
– inr-000, a cisco GSR 12008, can handle multiple GE interfaces, but at great expense, and the 12008 is already being “sunsetted” by cisco.
• requirement for redundancy means at least two routers
• need to maintain separation of ISP from non-ISP traffic would have require (at least) four border routers, rather than just two…
• … or use “Policy Constrained Routing” developed in Internet2 Routing WG.
• RFP for new campus border routers led to selection and purchase of two Juniper M40e routers– being installed and configured now, in full production Summer
2004EE122, 7 May 2004 20
Rec Sports CEV
Evans Hall
HPR
DC
ISP
HPR
DC
ISP
HPR+DC
ISP (pri)
ISP (sec)
ISP (pri)
ISP (sec)
mirrorport
mirrorport
shaper 1
shaper 2
resnetshaper
HPR+DC
HPR+DC
HPR+DC
CalREN
affiliate
cogent INR-668
affiliatesborderrouter
borderrouter
#2to
CalREN
campusnetwork
resnetINR-205
6500
INR-201
6500
INR-202
6500
akamaicluster
borderrouter
#1to
CalREN
3
1
2
04
567
8
3
1
2
0
4
56
7
8
MFN fiber toEmeryville
9
9
Figure 2. UC Berkeley Border Topology showing connections to next-generation CalREN network.
CENICcatalyst
4003
SBC GEservice to SF
UCB connectivity to CalREN (2)UCB connectivity to CalREN (2)
EE122, 7 May 2004 21
Policy Constrained RoutingPolicy Constrained Routing• developed by Cisco and Juniper, working with CENIC
and Internet2 Routing Working Group to address limitations of traditional destination-based routing decisions
• uses “virtual routing and forwarding” tables (VRFs) to make forwarding decisions based on router input interface and destination IP address– a specific VRF is applied to all packets entering a particular
interface– different VRFs, expressing different routing policies, applied to
different interfaces
• needed at UCB to maintain separation of ISP vs non-ISP traffic
EE122, 7 May 2004 22
Policy Constrained Routing in UCB border routersPolicy Constrained Routing in UCB border routers
HPR
DC
ISP
5
6
4
3
1
0
HPR+DCVRF
ISPVRF
2
8.0 8.1 8.2
ResNetPacketShaper
CampusPacketShaper
802.1Q subinterfaceson Gigabit Ethernet
AffiliateConnections
"ALL"VRF
INR-201
INR-202
INR-201
INR-202
INR-2057
UCBVRF
CalREN
A-1A-2
A-3
CalREN
UCB-ONLYVRF
EE122, 7 May 2004 23
National National LamdaRailLamdaRail• coalition of E&R entities, including CENIC, PNWGP, I2, Cisco
• uses DWDM (40 10GE lamdas) for multiple testbed networks– waves (L1) or switched ethernet (L2) or routed IP (L3
• first transcontinental Ethernet network
• additional dark fiber pairs