Operating Wide-Area Ethernet Networks Matt Davy Global NOC Matt Davy Global NOC.

Operating Wide-Area Operating Wide-Area Ethernet NetworksEthernet Networks

Operating Wide-Area Operating Wide-Area Ethernet NetworksEthernet Networks

Matt DavyMatt DavyGlobal NOCGlobal NOCMatt DavyMatt DavyGlobal NOCGlobal NOC

OutlineOutlineOutlineOutline

Overview of Networks

Configuration

Troubleshooting/Monitoring

Overview of Networks

Configuration

Troubleshooting/Monitoring

Overview of NetworksOverview of NetworksOverview of NetworksOverview of Networks

NLR FrameNet

nationwide ethernet over dwdm

18 Cisco 6509 switches

10GbE backbone

p2p and multipoint vlans

dedicated and best effort

NLR FrameNet

nationwide ethernet over dwdm

18 Cisco 6509 switches

10GbE backbone

p2p and multipoint vlans

dedicated and best effort


I-Light

Indiana’s statewide higher ed network

statewide ethernet over dwdm

19 Cisco 6509 switches (layer2 & layer3)

10GbE backbone with p2p vlans

I-Light

Indiana’s statewide higher ed network

statewide ethernet over dwdm

19 Cisco 6509 switches (layer2 & layer3)

10GbE backbone with p2p vlans


MANLAN

ethernet exchange in new york city

Cisco 6513 switch

1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn

local and wide-area connections

MANLAN

ethernet exchange in new york city

Cisco 6513 switch

1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn

local and wide-area connections


Indiana University campus network

large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses)

Cisco 6500’s and HP Procurve

very interesting stp design

~1,500 total switches

Indiana University campus network

large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses)

Cisco 6500’s and HP Procurve

very interesting stp design

~1,500 total switches

Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues

configuration of vlans

very manual and time intensive (manual = error prone)

need to automate this process

various control plane projects are one option, but could use something more lightweight

could use vtp ?

configuration of vlans

very manual and time intensive (manual = error prone)

need to automate this process

various control plane projects are one option, but could use something more lightweight

could use vtp ?


VLAN ID Assignment

big problem when interconnecting multiple layer2 domains

does Q-in-Q solve this ?

does vlan id translation solve this ?

VLAN ID Assignment

big problem when interconnecting multiple layer2 domains

does Q-in-Q solve this ?

does vlan id translation solve this ?


Q-in-Q

sounds good, but not flexible enough

want to map some .1q tags to outside vlan and want other .1q tags to get switched normally

customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in-Q.

also not implemented in all switches

Q-in-Q

sounds good, but not flexible enough

want to map some .1q tags to outside vlan and want other .1q tags to get switched normally

customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in-Q.

also not implemented in all switches


VLAN ID Translation

could help, but limitations in currently implementation

each port needs it’s own translation table

on 6500, translation table is shared across multiple ports

greatly confuses cross-domain troubleshooting

VLAN ID Translation

could help, but limitations in currently implementation

each port needs it’s own translation table

on 6500, translation table is shared across multiple ports

greatly confuses cross-domain troubleshooting


loops and spanning tree fun

spanning-tree is often not well understood

some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology

often does not help anyway when multiple layer2 domains are interconnected

loops and spanning tree fun

spanning-tree is often not well understood

some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology

often does not help anyway when multiple layer2 domains are interconnected


things that might help some:

enable spanning-tree within your domain

filter bpdus at the edge of your domain

limit total broadcast traffic on every port

make sure config has enough granularity for port speed (1% of 10G is still too much)

things that might help some:

enable spanning-tree within your domain

filter bpdus at the edge of your domain

limit total broadcast traffic on every port

make sure config has enough granularity for port speed (1% of 10G is still too much)

Loops Outside of Your Domain

#1 #2


why will a loop outside your domain hose your switch ?

not 100% clear

one possibility is mac address learning overload

switch flooded with packets for which it has to learn source mac addresses

mac addresses quickly flip-flop between ports

why will a loop outside your domain hose your switch ?

not 100% clear

one possibility is mac address learning overload

switch flooded with packets for which it has to learn source mac addresses

mac addresses quickly flip-flop between ports


how could this be avoided ?

turn off mac address learning

for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway

could also have out-of-band mechanism to statically configure mac forwarding tables

will this entirely protect you ? don’t know

how could this be avoided ?

turn off mac address learning

for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway

could also have out-of-band mechanism to statically configure mac forwarding tables

will this entirely protect you ? don’t know

Troubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/Monitoring

how can you tell when a vlan is down ?

hint: think break in the middle of the topology

on vlan trunks, can’t see how much traffic is associated with each vlan

CoS hack on the 6500’s for this

lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking

how can you tell when a vlan is down ?

hint: think break in the middle of the topology

on vlan trunks, can’t see how much traffic is associated with each vlan

CoS hack on the 6500’s for this

lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking

Troubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/Monitoring

tools to trace current vlan path across the network

IU has developed a spanning-tree mapping tool that helps with this

“turn-around interfaces” useful for debugging performance problems

tools to trace current vlan path across the network

IU has developed a spanning-tree mapping tool that helps with this

“turn-around interfaces” useful for debugging performance problems

Thank You Thank You Thank You Thank You

Operating Wide-Area Ethernet Networks Matt Davy Global NOC Matt Davy Global NOC.

Documents

Transcript of Operating Wide-Area Ethernet Networks Matt Davy Global NOC Matt Davy Global NOC.