Post on 28-Mar-2015
December 2007
Neil FrancisNetwork Administrator
SWERN
University of Bath
JANET Optical Event
SWERN rebuilt using Ethernet50 VLANS and 3 Telcos
2
SWERN2 - Outline
SWERN
One of the RNOsMain University connections
Bath, Bath Spa, Bristol, UWE, Exeter, Plymouth, Falmouth
Approx. 40 Further Education InstitutionsA dozen or so other institutionsAbout 450,000 users
3
Swern2 - Some Background
SWERN is in the South West of EnglandMerger of SWAN and BWEMAN
Essentially 2 networks connected togetherPiecemeal build – varied active equipmentSome resilience (always a worry)
Mixture of ATM, POS, Ethernet etcLimited HR
Two part time engineers, network managers, some Ops support etc
Geographical issues – long and thin, rural(ish)13 POPs – some remote
The usual stuff.....
5
SWERN2 - Opportunities
For SWERN 2 we had an opportunity to tidy things upThe usual design factors
Homogeneous, Resilient, Cost Effective?, Engineering, HR.
The South West is long and thinWe had POPs in the large conurbations
CheltenhamSwindonBathBristolTauntonExeterPlymouth
8
SWERN2 - Options
OptionsDark fibre, managed solutions etcAll considered but not only cost ruled these out.
Long distance ethernet circuits becoming more comfortable for usConcerns were things like
ReliabilityResilienceScalability (bandwidth)QOSSecurityUnderstanding exactly what we were getting
9
SWERN2 - Ethernet
Ethernet – but not quiteTypes of connection:
Point-point, point-multipoint, multipoint-multipoint, etc.Our equipment:
RoutersSwitches
Architecture: VPWS - Virtual Private WiresVPLS - Virtual Private LANsEthernet over Optical
etc...
Service Characteristics: VLAN MultiplexingVLAN TransparencyL2 PDU Transparency
10
SWERN2 - Ethernet
Telcos had solutions availableWe already had some metro ethernet cctsTail circuits only
Ethernet on the backbone meant...Bandwidth - fast, cheap, tunableStandard simple equipmentStandard circuit presentations (fibre and copper)Layer 2 links – easier to manageJust use point to point vlansWe do layer 3
It looked OK
11
SWERN2 – Ethernet Backbone
Connecting these all togetherConnect to enable loops
Two POPs in each cityConnect these together
Fairly close together so relatively easy
Connect each POP back to an RNEPUse Metro Ethernet circuits
Connect the RNEPs togetherFairly easy
End up with something like this....
12
SWERN2 – Ethernet Circuits
Essentially 6 loops here
Connect to standard ethernet equipment...
13
SWERN 2 – Standard Equipment
Cisco 7609 OSR RoutersSup. Engine 720/1Gbyte RAM Located at RNEPs
Cisco 7206VXR RoutersNPE-G1 Processors
Cisco Catalyst 494848 Port 10/100/1000
Ethernet Switch
14
SWERN 2 – Loops & VLANS
SJ5
SJ5
RNEPsPOPsEnd Sites
Allow only specific VLANS across the trunks
Trunks
15
SWERN2 - VLANS
16
SWERN2 - Tender
Tender issued late 2005Solutions based on mixture of
WDMSDH/SONET/EPLLES/WESMPLS
But all presented to us as ethernetCarrier EthernetMetro EthernetEvolved EthernetNational Ethernet
etc......
17
SWERN2 - Bandwidths
All bandwidths uncontended
18
SWERN2 - Engineering
With 6 loops 8 VLANS per loopPlus some test VLANS
Over 50 VLANS – all terminate on our RNEP routersAll traffic flows through these boxes
All point to point VLANS
Plus a management VLAN
We do the layer 3 stuff (OSPF, BGP)
We still have to manage, monitor, administer, troubleshoot etc.
19
SWERN 2 - Routing
20
SWERN 2 – Routing and Resilience
OSPFSingle small backbone (Area 0)Stub areas from the POP areasTimers tuned low
Also looked at BFD for fast link failure detectionBest case scenario – as good as POS!Worked between our POP routers and RNEPsNeeded 12.4.x IOS on our 7200s
Experimented with timers but were finding even a flash disk insertion was causing BFD to baulk
We currently rely on fast converging OSPF
21
SWERN2 - Observations
We know we have a mixture of the following:
Simple LES/WES circuits
DWDM/SDH/EPL circuits
EoMPLS circuits
We run point to point VLANS across all this technology and it works
Seemed too easy in many waysJust 2-3 people doing all this
22
SWERN2 - Observations
Management VLANSpanning tree should be turned onHave had more than one occurrence of loopingSomewhere - BPDUs are not being passedTaking longer to fix than it should
Witnessed CDP neighbor inconsistenciesTracked to provider switches being at different OS levels. A bug here but related to above.
Lost our .ad/.ah port configuration3 VLANS downProvider insistent all was workingFixed at 3rd level support – too long
23
SWERN2 – EPL Latency
24
SWERN2 – MPLS Latency
25
SWERN2 – Next stages
10 Gbps links across BristolConnecting RNEPS
Dark Fibre and 10GE over Optical
Still presented as EthernetTerminate on 10GE switchesProbably do some routing on same devicesDirect connection to RNEP routers10G POS to Janet Core
Everything else the same
26
SWERN2 - Possibilities
The concept of SWERN2 is all very easyIdentical POPs and equipmentIdentical circuit connections
Ideally we would want less POPs
Possibility to use Telco infrastructure to backhaul everything to our RNEPS
Tail circuits to institutions not an issue any moreResilienceQoS
28
SWERN3 - Maybe
Can't keep up with the service offeringsEthernet over SONETMPLS/VPLST-MPLS (ITU G.8110)802.1ad, 802.1ah, 812.1Qay802.3ah – Ethernet in the first mile
Etc...
Management802.1ag – Connectivity Fault Management802.1ag – End to End OAM/CFM
Management - is this the important bit?
29
SWERN2 - Observations
30
Any Comments
What we had
What we did
How we did it
Would anyone else do it this way?
What do other people do?
Observations