4. Data Center Networking Taking Risk Away From Layer 2 Interconnects

8/13/2019 4. Data Center Networking Taking Risk Away From Layer 2 Interconnects

1/93

2009 Cisco Systems, Inc. All rights reserved. Cisco PublicPresentation_ID 1

Ciscos DCI

Data Center Networking: Taking Riskaway from Layer 2 Interconnects

BRKDCT-2840


2/93

2009 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKDCT-2840 2

Objectives

Overview of Ciscos Data Center Interconnect Solutions

Understanding the need to Extend the Layer 2 DomainAcross Data Centers

Understanding Problems created due to extending Layer2 Domain Across Data Centers

Understanding the Recommended solution approaches

to solve today L2 Extension issues

Overview longer term solutions


3/93


Drivers for Data Center Interconnect

Core

Aggr/

DistrAccess

L3

L2

WAN

Data-center

WAN

Core

Aggr/

DistrAccess

L3

L2

WAN

Data-center

SAN SAN

Business Need

Disaster Prevention

Business Continuance

Workload mobility

IT Solutions

Active/Standby Migration

Server HA clusters, Geo-clustering

Move, consolidate servers, Vmotion

DWDM/CWDM


4/93


Business Application Resiliency

Business Resilience

Continued Operation ofBusiness During a Failure

Disaster Recovery

Protecting Data Through OffsiteData Replication

and Backup

Business ContinuanceRestoration of Business

After a Failure

Zero Down Time is the ultimate goal


5/93


Applications Classification

Which Vlans needs to be extended between Data Centers ?

VIPs

Network Services(ACE, FWSM)

Geo-Clusters(Veritas, MSCS, Oracle RAC, etc)

Vmotion Vlans(for ressource offloading and utilization)

Application Migration(both physical and virtual)

A/S A/A


6/93


Internal

Network

Active/Active

Application Processing

Active/Standby

Database Processing

Or

Active/Active

Internal

Network

Active/Active Web

Hosting

Active/Active Data Centers

InternetService

Provider A

Service

Provider B

Choice 1: Application session IP address change HTTP Redirect

DNS Based

L3 /32-routing with Route Health Injection (RHI)

Choice 2: Application session IP address unchanged Subnet extension

Can IP achieve Active/Active ?

Per server Site selection Hot standby server protection


7/93 2009 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKDCT-2840 7

Network HA & Applications HAImplications in regard of the network technology used

Application Resilience

Network Resilience(stability, convergence time)

Time

EvolutionL2

STP

L2

STP BP

VSS or VPC OTVVPLS

OTV

+ TRILL

L3

routing

HOTw/ total

DC independance

+ internal DC resilience

High-Availability

WARM

HOTw/ DC coupling

HOTw/ DC CP

independance

HOTw/ total DC

independance

= isolated L2

=L2oL3



The key middleware for Business continuance isHA Cluster/GeoCluster

* Microsoft MSCS

* Veritas Cluster Server (Local)

* Solaris Sun Cluster Enterprise

VMware Cluster (Local)

Oracle RAC (Real Appl.Cluster)

HP MC/ServiceGuard

HP NonStop

HP Open VMS/TruCluster

IBM HACMP

EMS/Legato Automated Availability Mgr

* Veritas offers an extended Cluster solution using L3 for

inter-site connectivity

Microsoft Windows Server 2008 supports L3 site to site

(def=IPv6).

Sun Geographic Framework Edition



DC 1 DC 2

Production Network

Virtual Center

L2 extension for VMotion Network

The key middleware for Workload mobility is

Virtual Mach ines and Vmo t ion

ESX-A source ESX-B target

Core

Switch

Core

Switch



DataCore

DC 1 DC 2

Production Network

Virtual Center

L2 extension for VMotion Network

VMotion Requirements using SAN motion

ESX-A source ESX-B target

Synchronous replication

Virtual Disk

Primary path Alternate pathPrimary pathAlternate path

~100 kms max

Core

Switch

Core

Switch



VMWARE

VMware Virtual Desktop

VMWARE VMWARE VMWARE

VMWARE INFRASTRUCTURE

Centralized desktop

management

Connexion

broker

Thin client Virtualdesktop



Business benefits

For Enterprise:

Business continuity with hot standby

Flexible integration of Service Delivery Centers

Cost reduction For Service Provider or outsourcer:

Delivery of new Data Center services for enterprises

Smooth integration of enterprise applications with SaaS

For OTT

Delocalization of Service Delivery Centers in Service ProviderData Centers



Datacenters Interconnect (DCI) Considerations

L2 CoreL2L2

Main Data

Center

Backup

Data Center

StorageStorage

IP core

DWDM/CWDM

L3L3

SAN SAN

L2L2

FC FC

WAASWAAS

VLAN extension

SAN extension

Layer 3 extension

Network services extension

DCI Involves



HA Clustersystem

Network & SecurityServices required

No servicerequired

HA Clustersystem


No servicerequired

HA Clustersystem


No servicerequired

The L2 DCI modelCreate extended VLAN without extending Spanning-tree

Control broadcast domain



DCI VLAN extension key technical challenges

L2 control-planeSTP domain scalability

STP domain isolation

L2 Gateway redundancy

Inter-site transport

Long distance link protection with fast convergencePoint to Point & Multi-points bridging

Path diversity

L2 based Load repartition

Optimized routing egress & ingress

Extension over IP cloudMulticast optimization

L2 data-planeBridging data-plane flooding & broadcasting storm control

Outbound MAC learning

Technology challenge:

L2 is weak

IP is not mobile



Cisco Data Center InterconnectSolutions

P2P extension MAC Bridging MAC routing

Cat 6500 VSS Cat 6500 VSS HUB

w DWDM optics

N7K(OTV)N7KvPC N7KvPC HUB w Optical

Device

TRILL (L2MP)

ASR + Cat 6500

(EoMPLS over GRE)

Cat6500) (VPLSoGRE)

Cat 6500 (VirtualEthernet )

N7K(OTV)

VSS- Virtual Switching System, vPC Virtual Port Channel, DWDM Dense Wavelength Division Multiplexing

EoMPLS Ethernet over MPLS, VPLS- Virtual Private LAN service,OTV- Overlay Transport Virtualization

ASR + Cat 6500(EoMPLS )

Cat 6500 + C7600

CRS-1 + ASR9K(VPLS)

N7K(OTV)

Transport Options

Fiber

MPLS

IP



Native Ethernet solutions



Positioning

STP isolation is performed by usage of Multi-ChassisEtherchannel

LACP / UDLD is now the link handling protocol

Perfectly adapted to dark fibers and protected DWDM

Requires Hub&Spoke interconnection design

MAC-based load repartition

Native Multicast replication


19/93 2009 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKDCT-284019

Multi-Chassis Etherchannel (MEC) SolutionsVSS, vPC

Both VSS-MECand vPC are a

Port-channelingconcept extendinglink aggregation to

two separatephysical switches

Allows thecreation ofresilient L2

topologies basedon Link

Aggregation.

Eliminates thedependence onSTP in the L2

access-distribution Layer

Scale AvailableLayer 2

Bandwidth

Simplify NetworkDesign

Virtual Port Channel (vPC)

Non-VPC vPC

L2 SiSi

Virtual Switching System (VSS)

Non-VSS VSS

SiSi

Catalyst 6500 Nexus 7000

L2



2 Server PODs

High link utilization with MEC

6 New Links for POD

Interconnect DC Core not necessary

Dual Sites interconnectionIts Really a Question of Scale and Manageability

DCI

DCI point is

STP isolation (BPDU filering)

Broadcast storm control



4 Server PODs with Core Tier

Easy to add more PODs

Fewer links in the core

Easy bandwidth upgrade

Switch peering complexityreduced

Predictable performancethroughput, latency,convergence, etc..

Multi-Sites interconnection with coreIts Really a Question of Scale and Manageability

DCI point is

STP isolation (BPDU filering)

Broadcast storm control



Layer 2 Extension using VSS, vPC over DarkFiber Multi Site

VSLAggregation

VSL VSLSiSi

VSL, vPC

Access

SiSi

Switches use separateLambda to

Interconnect

Switches use separateLambda to

Interconnect

DWDM CORE

SROptics

SROptics

Physical View

SiSi

DC1 DC4DC2 DC3

SiSi SiSiSiSi

VSS

N7K

VSS

N7K

MEC MECMEC MEC



VSS / vPC Data-Center InterconnectScaling validation testing

VSL or vPC Peer Link extended over 100km fiber

Layer 2:

200 / 500 Layer 2 VLANs

100 VLAN SVIs

10,000 client-to-server flows

20 Gbps traffic flows between data centers

Layer 3

1000 BGP routes also redistributed to OSPF

+ 5000 OSPF routes

Results: L2/L3 Unicast & Multicast traffic protected on any failure in

VSS = 2.2s worst case

vPC = 2.8s worst case some specific case at 5s

Storm control contained on failing site



Encrypt

on egressinterface

Decrypt

Cisco TrustSec Link-layer cryptographyHop -by -Hop Packet Con fident ial i ty and Integrity v ia IEEE 802.1AE

Bump-in-the-wire model

Packets are encrypted on egress

Packets are decrypted on ingress

Packets are in the clear in the device

Allows the network to continue to perform all the packet inspection featurescurrently used

Can be incrementally deployed depending on link vulnerability

Nexus: Wire-rate link-layer encryption on every 10/100/1000/10GbE port

Decrypt

on ingressInterface

Encrypt

Packets in the clear inside the system

802.1AE Encrypted 802.1AE Encrypted802.1AE Encrypted

cipher dataIn the clearcipher data In the clear cipher data


25/93


Label Based transportover MPLS & over GRE


26/93


L3 transport benefits for L2 interconnect

Several main improvements can be achieved using L2oL3 versus

simple bridging

Inter-DC link L3 protection

Core STP / LACP / UDLD suppression

Core links are protected via L3 convergence

Fast detection / Dampening

Stability & Fast-convergence

Emulated circuit for Point to Point

EoMPLS for dual site connection

Easy link sharing between L2 & L3

Inter-DC Spanning-tree suppression using VPLS

each DC STP will be isolated from each others

L2 Fault domain is restrained

(Dual si tes u sing EoMPLS / Mult ip le si tes us ing VPLS)


27/93


EoMPLS Port Mode

MPLS

802.1QCross-connect

Transparent to Edge bridging(BPDU, SPT, VLAN, CoS)

FRR / TE / LB802.1Q

Cross-connect

Back-up

Interface Giga n/n

switch mode type Access or Trunk


28/93


STP isolation over EoMPLS

Site A Site B

PROS: Native STP isolation, load balancing,Fast convergence

Notice: Requires Remote Ethernet PortShutdown

6500

orASR1K

Etherchannel over EoMPLS Etherchannel over VSS-Aware EoMPLSSite A Site B

EoMPLS device is VSS

Available SUP-720B with 12.2(33)SXI.2

6500 VSS

PW-Redundancy over EoMPLS

Local

STP

Local

STP

Agg1 Agg2 Agg1 Agg2

Acc1 Acc2 Acc1 Acc2

Site A Site B

EoMPLS PW-redundancy

Requires EEM script synchronization

6500

7600


29/93


Etherchannel from aggregation over EoMPLS

U-PEPrimary

LDP is ensuring PW backupfor core link or node failure

U-PEBack-up

U-PEPrimary

U-PEBack-up

VSS / VPC VSS / VPC

Standard LACP / UDLP is slow detection,

LACP fast-hellos can be used, but are subject to false detect

Best is to use remote ethernet port shutdown option (native with ARS1K, EEM script with

6500)

MPLSCore

(May be justdirect links)


30/93


EoMPLS Remote Ethernet Port Shutdownfor Catalyst 6500

xconnect logging pseudowire statusevent manager applet PseudoWire-101-Down

event syslog pattern "MPLS peer 10.127.127.2 vcid 101, VC DOWN

action 1.0 cli command "enable

action 2.0 cli command "conf t

action 3.0 cli command "int gi2/7

action 4.0 cli command "shut

action 5.0 cli command "no shut

action 6.0 syslog msg "Pseudowire 101 Down

Rem: Native with ASR1K

http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/12s_resd.html.

ip routing protocol purge interface

router ospf 1

timers throttle spf 10 100 5000

timers throttle lsa all 10 100 5000
http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/12s_resd.htmlhttp://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/12s_resd.html


31/93


Virtual Private LAN Service (VPLS)

VPLS defines an architecture that allows MPLS networks to offerLayer 2 multipoint Ethernet Services

Metro Core emulates an IEEE Ethernet bridge (virtual)

Virtual Bridges (VFI) linked with Pseudo Wires

PE PECE CE

VPLS Multipoint Services

CE

MPLSVFI VFI

VFI

PW


32/93


VPLS L2 signalling and forwardingaka Transparent-Bridg ing

A

B

C

VFI VPN 1

-

VFI VPN 1

-

VFI VPN1

-

AB

Ea

VCID

333

VCID

111

VCID

222

- Ea : A

AB

AB

- VCID 111 : A

- VCID 333 :A

AB

AB

BA

Eb- Eb : B

BA

- VCID 111 : BBA


33/93


VPLSoGRE

VPLS connectivity over IP-only network.VPLS PW are established over MPLSoGRE Tunnels.

Requires SIP on the Catalyst 6500

PE PECE CE

VPLS Multipoint Services

CE

VFIVFI

VFI

IP

GRE


34/93


VPLSoGRE (without encryption):Ethernet MTU = 1576 bytes max

Core link Ethernet encapsulation:

+ Core Ethernet header = 14

+ Optional core 802.1Q = 4 (could be null when no core VLAN)

+ Core Trailer (FCS) = 4

GRE encapsulation: (24)

+ IP header = 20

+ GRE encaps = 4

MPLS encapsulation: (4-16)

+ Core LDP = 4 (could be null when direct link)

+ Targetted-LDP (VPN-id) = 4

L2VPN: (18-22)

+ AToM options = 4 (not optional in L2 mode)

+ Ethernet (DA/SA/Type) = 14

+ Optional edge 802.1Q header = 4 (when H-VPLS or EoMPLS type4)

(rem: no edge FCS encapsulated)

PDU = 1500


35/93


VPLS DCI solutions summary

6500SIP-400

6500SIP-600

VSSSIP-400

7600ES / ES+

ASR9K CRS-1

VPLS

VPLSoGRE

PE

redundancywith STP

isolation

1

1

2

3

4

4

1 = Using EEM Semaphore (since Q1CY08)

2 = Virtual-Ethernet native (Q1CY10)3 = MC-LAG native (since Q4CY09)

4 = Using EEM Semaphore at aggregation level (Q3CY10)

rem: MC-LAG planned for IOS-XR V4.x (CY10)

1

1 S


36/93


CVD Validated solution

for Cisco 7600 & Catalyst 6500using scr ip ted s emaphore protoco l s ince 01CY08

Root

LocalSTP

N-PE

Primary

N-PE

Back-up

MPLSCore

Semaphore

Trigger

Edge STPMAC flush

Edge port

Edge port

PWMAC withdraw

Cisco Validated Design VPLS access control executed by EEM

http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/desguide.pdf

http://www.ciscopress.com/bookstore/product.asp?isbn=1587059924

1 - EEM Semaphore

1 EEM S h
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/desguide.pdfhttp://www.ciscopress.com/bookstore/product.asp?isbn=1587059924http://www.ciscopress.com/bookstore/product.asp?isbn=1587059924http://www.ciscopress.com/bookstore/product.asp?isbn=1587059924http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/desguide.pdfhttp://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/desguide.pdf


37/93


MPLS

Core

Spanning Tree Isolation- EEM in N-PE

LDP is ensuring PW backup

for core link or node failure

B signals Backup is down

Primary PW is set up

P signals Primary is UP

Backup PW is forced down

B is set down

N-PE

Primary

N-PE

Backup

N-PE

Primary

N-PE

Backup N-PE

BackupN-PE

Primary

Root

Local

STP

Root

Local

STP

Root

LocalSTP

P signals Primary is Down

Backup PW is forced UP

B is set UP immediately

P semaphore is failing

forcing backup mode

B signals Backup is active

Primary PW is maintained

down

P signals Primary is up

wait a 60s start-up delay

P semaphore is up, but

B is still up for a delay,

forcing backup mode

P signals Primary is up

wait a start-up delay (60secs)

Backup PW is forced down

B is set down

B signals Backup is down

Primary PW is set up

P semaphore is up since

startup-delay, B is forced

down.

Nominal modePrimary N-PE failure modePrimary N-PE Operational mode

1 - EEM Semaphore

1 EEM S h


38/93


EEM-semaphore adapts toMultiple DC topologies

Root

LocalSTP

N-PEPrimary

N-PE

Back-up

MPLSCore

RootN-PEPrimary

N-PEBack-up

MPLSCore

LocalSTP

Option N3 H-VPLSOption N2 VPLS

Root

LocalSTP

N-PEPrimary

N-PEBack-up

MPLSCore

Option N4

Multi-domains H-VPLS

VSS / VPCN-PEPrimary

N-PEBack-up

MPLSCore

Option N5

Multi-domain H-VPLS with MEC

sub-2s convergence on any failure

1 - EEM Semaphore

2 Vi t l Eth t


39/93


The Virtual Ethernet Solution

nPE

Agg

Agg

nPE

VSS system

Agg

Agg

AggAgg

VSL VSL

VSS system

L2/L3/L4 LB

between all sites

Want to add a 3rd site?

VSL

Split horizon between

all neighbors for loop

avoidance

2 - Virtual-Ethernet

Th Vi t l Eth t S l ti 2 Vi t l Eth t


40/93


The Virtual Ethernet SolutionVPLS Conf igurat ion

pseudowire-class cl1

encap mpls! enable ML PW (ECMP LB)

load-balance flow

! enable FAT PW

flow-label enable

!

interface virtual-ethernet 1

! transport configuration

transport vpls mesh

neighbor 2.2.2.2 pw-class cl1

! service configuration

switchport

switchport mode trunk

switchport trunk allowed vlan 10,20

IP/MPLS

PE1 (1.1.1.1)

PE2 (2.2.2.2) PE3 (3.3.3.3)


Th Vi t l Eth t S l ti 2 Vi t l Eth t


41/93


The Virtual Ethernet SolutionVPLSoGRE Conf igu rat ion

int tunnel 1

tunnel mode gre

mpls ip

tunnel source 11.11.11.11

tunnel destination 22.22.22.22

tunnel route-via Gi1/1/1

!

int tunnel 2

tunnel mode gre

mpls ip

tunnel source 11.11.11.12


tunnel route-via Gi1/1/2

!

pseudowire-class cl1

encap mpls

! enable ML PW (ECMP LB)

load-balance flow

!

interface virtual-ethernet 1

! transport configuration

transport vpls mesh

neighbor 2.2.2.2 pw-class cl1

! service configuration

switchport

switchport mode trunk

switchport trunk allowed vlan 10, 20

Ip route 2.2.2.2 255.255.255.255 Tunnel1

Ip route 2.2.2.2 255.255.255.255 Tunnel2


C t l t 6500 SIP 400 f l i


42/93


Catalyst 6500 SIP-400 performance analysisNative VPLS interconnection

n * 1GE

n * 1GE

Customer switch LAN SIP-400

IEL CPE

dot1Q

with up to 7 VLANs on

up to 3 * 1GE port

with shaping/queuing

Line rate at 128Bytes

VPLS N-PE

IEL CPE

IEL CPE


43/93


Catalyst 6500 SIP-400 performance analysisVPLS o GRE o VTI in one box w ith w rap-cable

GRE

Bridged

LAN port

SIP-400

H-QoS

SIP-400VPLSoGRE H-QoSSIP-400

VRF

edge

VRF

core

At 2Gbps FDX (bi-directional):

no drop occurs into Real-Time queue at 192Bytes MTU

6% drops into RT for 128Bytes

50% drop into RT for 64Bytes

SSC-600

(VTI)

VF

I

VLAN

or

QinQ

GRE

GRE

One GRE per destination site


44/93


45/93


QinQ caveat to be aware of

QinQ is key to scalability until 802.1ah

QinQ usage is presenting a caveat for virtual MAC-addresses

A

A

A

AA

Avoid usage of same mac-address for Virtual-MACMainly HSRP

Avoid FW or ACE Active/active state extension Control MAC-add setting into Virtual-Machines

configure no mac-learning on PE edge ports


46/93


EFPs

MPLS

H-VPLS Architecture:

802.1ah Flexible Forwarding Model

TRUNK

L3

L2

L2

P2P XCONNECT

EVC to L3/VRF

C-BRIDGE

B-BRIDGE

L2BRIDGED

Local Connect

EFPs

3 - MC-LAG


47/93


N-PE Active/Standby conceptsusin g Mult i -Chassis Link Aggregat ion Group MC-LAG

LocalSTP

7600

Primary

7600

Back-up

MPLSCore

ICCP

PWMAC withdraw

mLAC

P

draft-martini-pwe3-iccp

Inter-chassis Control Protocol (ICCP) is an LDP based hello protocol for node clustering

One only side active per MEC at first phase

VSS/vPC

7600 - 12.2.(33)SRE

- MC-LAG

3 - MC-LAG


48/93


N-PE Active/Standby conceptsusin g Mult i -Chassis Link Aggregat ion Group MC-LAG

Primary

Back-up

MPLSCore

ICCP

PWMAC withdraw

MC-LA

G

draft-martini-pwe3-iccp

Inter-chassis Control Protocol (ICCP) is an LDP based hello protocol for node clustering

One only side active per Multi-Chasis Ether-Channel at first phase

MCEC

MC LAG


49/93


FAT label load balancing effect VC ID Based Load Balancing may result in very uneven load split

VID 100

MAC A->B

VID 100

MAC A->B

VID-100

MAC C->D

VID 200

MAC A->B

VID-200

MAC C->D

Member Link 1

Member Link 2

FAT PW Load Balancing flows split across the member link and core

VID 100

MAC A->B

VID 100

MAC A->B

VID-100

MAC C->D

VID200

MAC A->B

VID-200

MAC C->D

Member Link 1

Member Link 2

SA DA DATAL2

Header

MPLS

Label

MPLS

Label

MPLS

Label

VC

Label

FAT

Label

Bottom label includes FAT

label which allows per flow

load balancing across the

network. Single flow followssingle path

4 - EEM Semaphore


50/93


MPLS based DCI Solution with CRS-1

Collapse WAN/Core layer with CRS

WAN/Core Router Requirements:

Connectivity options(Nx10GE, 1GE, POS OC-X)

NPE Redundancy/failover

Site/Cloud transparency

L2/L3/Storage extension

Sub-rate QoS

Scale, path steering

STP isolation and loop avoidance

Routing Scale

Roadmap to 40/100GE Capacity

Access

Core

Aggr/Distr

WAN

MPLS

Collapse

WAN/Core

layers

30~60

10GE

density,

with POS

EEM Semaphore


51/93


52/93


MPLS TE for link bundle balancing

Root

Local

STP

Root

Local

STP

Parallel TE tunnels

Parallel TE tunnels

For backup path

Selective QinQ

(or multiple Q-links)

T ffi E i i C fi ti


53/93


Traffic-Engineering ConfigurationPush PW into TE-Tunnels

interface Tunnel16

ip unnumbered Loopback98

mpls ip


tunnel mode mpls traffic-eng

tunnel mpls traff path-option 10 explicit name LB-Primary

tunnel mpls traff path-option 20 explicit name LB-Secondary

ip explicit-path name LB-Primary enable

next-address 192.169.14.4

pseudowire-class VPLS-Tunnel-16

encapsulation mpls

preferred-path interface Tunnel16

!

l2 vfi VFI-99 manual

vpn id 99

neighbor 10.98.76.6 pw-class VPLS-Tunnel-16

Have the VFI using the alternate TE

tunnel


54/93


Review of label based solutions

1. DCI executed in VPLS PE- With None or very few requirement toward aggregation

today on Catalyst 6500 & Cisco 7600

Cisco Validated Design (CVD) for semaphore protocol script

- With Multi-Chassis Ether-Channel toward aggregation

upcoming

EoMPLS VSS support (Q3CY09 under validation)

Cisco 7600 native MC-LAG in SRE release (Q4CY09)

Catalyst 6500-SIP400 Virtual-Ethernet with VSS in Q1CY10

2. DCI executed in aggregation (N7K / Catalyst 6500)

- For any VPLS core

today

N7K semaphore protocol script under validation over CRS-1 core

Under work also for Cat6k edge or for ASR9K core


55/93


Storm Propagation

All DCI approaches are solving

L2 Control-plane isolation (STP Isolation)

VPLS is solving:

Link quality with L3 protection

VPLSoGRE is in addition solving a third problematic:

L2 extension over IP core

But none of them is solving Data-plane storm control


56/93


Data-Plane storm control

When a DC becomes crazy, due to local STP failure,five types of traffic have to be considered

L2 control-plane

L2 broadcast

L2 multicast L2 known unicast

L2 unknown unicast (UU)

Storm is huge and permanent as L2 frame do not have

any TTLSome are hitting CPU, some just overflow links

Storm contol


57/93


Storm contolL2 broadcast and multicast packet storm

L2 broadcast

These packets are killing frames for switches when they reach CPU

Storm-control broadcast level xx,xx%

command must be installed on access facing link (I-Link or Q-link)

L2 multicast

These packets are killing frames for switches, if L3 multicast isenabled on CPU

Storm-control multicast level xx,xx%

command must be installed on access facing link (I-Link or Q-link)

In any cases, CoPP (Control Plane Policing) must be used to protect MSFC

against storm (at least ARP storm)


58/93


Routing consideration

Routing interaction with VLAN extension


59/93


Cluster VLAN D (L2 Only)

Cluster VLAN C (L2 Only)

Cluster Node A

Layer3 Core

Cluster Node B

VLAN A VLAN A

out g te act o t e te s oDefault Gateway Shared Between Sites

Data Center 1 Data Center 2

Egress traffic: When no Firewall at application layer,

FHRP gateway could be localized on site

Ingress traffic:How to attract traffic to site where server is localized ?

1) /32 dynamic announcement

2) LISP


60/93


HSRP group isolation ACL sample

mac access-list extended HSRP_MAC_VACL_Deny

permit 0000.0c07.ac00 0000.0000.00ff any

permit any 0000.0c07.ac00 0000.0000.00ff

permit 0000.0c9f.f000 0000.0000.0fff any

permit any 0000.0c9f.f000 0000.0000.0fffpermit any host 0100.5e00.0002

permit host 0100.5e00.0002 any

permit any host 0100.5e00.0066

permit host 0100.5e00.0066 any

!

mac access-list extended HSRP_MAC_VACL_Allow

permit any any

vlan access-map HSRP 10

match mac address HSRP_MAC_VACL_Deny

action drop

!

vlan access-map HSRP 20

match mac address HSRP_MAC_VACL_Allow

action forward

vlan filter HSRP vlan-list 3001, 3002, 3003

end

HSRPv1 Virtual MAC

HSRPv2 Virtual MAC

HSRPv1 Hello

HSRPv1 Hello

VRRP would be similar

HA Cluster access:


61/93


HA Cluster access:inbound traffic falls over to backup DC

ISP A ISP B

DC primary DC secondary

Ha_cluster_node1 Active

Cluster VIP 10.1.1.100

Ha_cluster_node2 Standby

RHIACE probe filtering

Public network

private network

Zzzz..

10.1.1.100 probe

failed10.1.1.100 probe

is OK

HA Cluster access:


62/93


HA Cluster access:inbound traffic falls over to backup DC

ISP A ISP B

DC primary

Public network

private network

Ha_cluster_node1 OFF Ha_cluster_node2 Active

Cluster VIP 10.1.1.100

RHIACE probe filtering

10.1.1.100 probe

is OK10.1.1.100 probe

fails

DC secondary


63/93


OTV

Overlay Transport Virtualization

For in depth OTV, please refer to BRKDCT-2001

run by Victor Moreno - Nexus 7000 TME


64/93


Overlay Transport VirtualizationTechnology Pillars

Protocol Learning

Built-in Loop Prevention

Preserve FailureBoundary

Seamless Site

Addition/Removal

Automated Multi-homing

Packet Switching

No Pseudo-Wire State

Maintenance

Optimal MulticastReplication

Multi-point Connectivity

Point-to-Cloud Model

OTV is a MAC in IP technique for

supporting Layer 2 VPNs overany transport.


65/93

Neighbor Discovery


66/93


Neighbor Discovery

Each Edge Device is adjacent to all the other Edge

Devices from the OTV Control Plane perspective.

OTV

Core

OTV

IP A

IP B

West East

IP C

South

OTV

Control

PlaneControl

Plane

ControlPlaneMulticast enabled CoreEdge Devices join acommon Multicast GroupAll signaling takes placeover the multicast groupMultipoint optimized

traffic replication

Non-multicast CoreEdge Devices register toan Adjacency ServerAdjacency list distributedto all participating devicesPoint-to-point unicastpeering for signaling



67/93


Eth 4

Eth 3

MAC TABLE

VLAN MAC IF

100 MAC 1 Eth 2

100 MAC 2 Eth 1

100 MAC 3 IP B

100 MAC 4 IP B

MAC 2

MAC 1


OTV Data Plane: Unicast

Core

MAC 4

MAC 3

OTV

IP A IP B

West East

L2 L3 L3 L2

OTV Inter-Site Traffic

MAC Table contains

MAC addresses reachable throughIP addresses

IP A IP BMAC 1 MAC 3

OTV

Encap

2

Layer 2

Lookup

1

No Pseudo-Wire state is maintained.

The encapsulation is done based on a destinationlookup, rather than based on a circuit lookup.

3 Decap4 MAC 1 MAC 3

6MAC 1 MAC 3IP A IP BMAC 1 MAC 3

MAC TABLE

VLAN MAC IF

100 MAC 1 IP A

100 MAC 2 IP A

100 MAC 3 Eth 3

100 MAC 4 Eth 4

Eth 1

Eth 2

Layer 2

Lookup

5

MAC 1 MAC 3



68/93



OTV Data Plane Encapsulation

OTV uses Ethernet over GRE encapsulation and adds an OTV shim to theheader to encode VLAN information.

The VLAN field of the 802.1Q header is copied over into the OTV header.

The overhead must be taken into account with respect to the MTU within thecore. Nothing new, VPLS has its own overhead.

DMAC SMAC Eth Payload

28 Bytes

overhead

6B 6B 2B 20B 4B 4B

DMAC SMACEtherType IP Header

Original Frame 4B

CRCGRE

HeaderOTV

Header

802.1Q

802.1Q

+14 (18) L2 Bytes

overhead



69/93


Source

OTV


Data Plane: Multicast

Core

Receiver

OTV

IP D

IP B

West

East

IP C

Receiver

IP ANorth

South

OTV

OTV

OIF-List

Group IF

G1 Overlay 1

Encap

2

Lookup

1

MAC 1 Mcast MAC

IP D SSM GMAC 1

Mcast MAC

3 Core

Replication

IP D SSM GMAC 1

Mcast MAC

OTV device perform IGMP snooping



70/93


Source

OTV



Core

Receiver

OTV

IP D

IP B

East

IP C

Receiver

IP ANorth

South

OTV

OTV

OIF-List

Group IF

G1 Overlay 1

Encap

2

Lookup

1

IP D SSM GMAC 1

Mcast MAC

3 Core

Replication

4

4

IP D SSM GMAC 1

Mcast MAC

West



71/93


Source

OTV



Core

Receiver

OTV

IP D

IP B

East

Optimal Multicast

ReplicationMulti-point Connectivity

IP C

Receiver

IP ANorth

South

OTV

OTV

OIF-List

Group IF

G1 Overlay 1

Encap

2

Lookup

1

MAC 1Mcast MAC

3 Core

Replication

MAC 1 Mcast MAC

Decap

5

The North Site did

NOT receive the packet!

6

6

4

4

IP D SSM GMAC 1

Mcast MAC

Decap

5

IP D SSM GMAC 1

Mcast MAC

West


72/93


73/93


STP BPDU Handling

When STP is configured at a site, an Edge Device will send andreceive BPDUs on the internal interfaces.

An OTV Edge Device will not originate or forward BPDUs on theoverlay network.

An OTV Edge Device can become (but it is not required to) a root of

one or more spanning trees within the site.

An OTV Edge Device will take the typical action when receivingTopology Change Notification (TCNs) messages.

OTV

Core

The BPDUs

stop here


74/93


Unknown Unicast Packet Handling

Flooding of unknown unicast over the overlay is not required and is

therefore suppressed.

Any unknown unicasts that reach the OTV edge device will not beforwarded onto the overlay.

The assumption here is that the end-points connected to the networkare not silent or uni-directional.

MAC addresses for uni-directional host are learnt and advertised bysnooping the hosts ARP reply

OTV

Core

No MAC 3 in the

MAC Table

MAC 1 MAC 3

MAC TABLE

VLAN MAC IF

100 MAC 1 Eth1

100 MAC 2 IP B


75/93


Unknown Unicast Selective Flood

Microsoft Cluster Services leverage unidirectional MAC addressesto force flooding to its cluster members

This flooding behavior is based on keeping a unicast MAC addressunknown by not sourcing any traffic from it

Multiple nodes will share the address in question

As traffic is flooded to this unknown address, all hosts received theflooded messages

OTV provides the ability to selectively flood traffic for specific MACaddresses in order to support this corner case.

Microsoft Cluster Services with NLB


76/93


77/93



OTV includes the logic necessary to avoid the creation of loops inmulti-homed site scenarios.

Each site will have its own STP domain, which is separate andindependent from the STP domains in other sites, even though allsites will be part of common Layer 2 domain.

Multi-Homing: Loop Condition Handling

Core

OTV

OTV

OTV

OTV

STP

domain 1

STP

domain 2No STP


78/93


79/93



A broadcast packet gets to all the Edge Devices within a site.

The AED for the VLAN is the only Edge Device that forwards broadcastpackets on the overlay network.

All the Edge Devices at a remote site will receive the broadcast packet, butonly the AED at the remote site will forward the packet into the site.

Once sent into the site, the packet gets to all switches on the site specificSpanning Tree.

Multi-Homing: AED & Broadcast

Core

OTV

OTV

OTV

AEDAED

Bcast

pkt

Broadcast

stops here

Broadcast

stops here

OTV


80/93


81/93


Guidelines & Limitations

Within a system/VDC a given VLAN can eitherbe associated with anSVI (VLAN interface) orextended using OTV.

As seen from the design section, this restriction does not pose a dramaticimpact to the feature deployment.

Overlay interfaces share the same site-VLAN

Only one external-interface can be specified. This interface is used tosource multicast traffic and attract traffic to the site. Unicast traffic sent toother sites is load-balanced based on routing table.

Design Option


82/93


Design Option

Access

Agg

Core

Layer 2 Link

Layer 3 Link

OTV Virtual Link

Pod A

WAN

Pod N

DCI DCI DCI DCI

OTV VDC as an appliance at the

Aggregation Layer.

PIM from the WAN-core reachingto the Aggregation Layer.

OTV VDC joins the mcastcore groups at the Agg

Layer.

Medium-to-Large Site:3-Tier DesignNo Dedicated DCI Connection


83/93


TRILL / L2 overviewPlanned CY10

L2 Multi-Paths


84/93

PRE-TRILL Network


85/93


Bridge Domain

CE

Cust. Eth Header

Cust IP Packet

802.1q Bridge

or HUB

Cust. Eth Header

Cust IP Packet

CE

CE1

CE

CE2

Links Disabledby Spanning Tree

Root

TRILL Network


86/93


RBridge Domain

CE

Cust. Eth Header

Cust IP Packet

Cust. Eth Header

Cust IP Packet

TRILL Header

RBridge

802.1q Bridge

or HUBNext Hop(1)

Eth Header

Cust. Eth Header

Cust IP Packet

TRILL Header

Next Hop(2)

Eth Header

Cust. Eth Header

Cust IP Packet

CE

CE

CE

CE

TRILL B i H ll


87/93


TRILL Basics - Hellos

ISIS Hellos are sent using a MAC of All-IS-IS-RBridges

Hellos are used for neighbor discovery and exchange ofinfo including:

RB System ID is 48 bits (typically MAC address)

All standard ISIS infoDesired Designated VLAN, Designated VLAN

Announcing VLAN Set, Forwarding VLAN Set

The RBridge Nickname (to save space)

RB Hello packets may be sent outside the RB network todiscover and eliminate external loops

Very Large Scale-out 10 GE Clusters


88/93


Using D1 Series Modules

16 Chassis

16 x10GE

Etherchannel

L2MP load-

balancing

across 16

chassis

256 10GE

hosts perswitch

32 Chassis

Up to 8,192 Servers Non-Blocking Using D1 Modules and L2MP

technology to design very large scale

10GE clusters

32 x 1/10GE Non-Blocking Ports per

module

Optimizes Nexus 7018 density

End of row or middle of row architecture

Scalability can be significantly greater

with 40/100 GE Uplinks

S


89/93


Summary

Data Center Interconnect Solutions allow redundant,scalable, secure Layer 2 VLAN extension

Catalyst 6500 VSS, Nexus vPC allow powerful and

simple DCI over dark fibers or protected D-WDM

MPLS based solutions are mature to handle SP based

Ethernet connections using VPLS

EoMPLSoGRE & VPLSoGRE are opening capability

over IP network

OTV will allow DCI directly from aggregation layer in a

very efficient and simple way

BRKDCT 2840 Recommended Reading


91/93


Questions ?

C l t Y S i E l ti


92/93


Complete Your Session Evaluation

Please give us your feedback!!

Complete the evaluation form you weregiven when you entered the room

This is session BRKDCT-2840

Dont forget to complete the overallevent evaluation form included inyour registration kit

YOUR FEEDBACK IS VERYIMPORTANT FOR US!!! THANKS


93/93

4. Data Center Networking Taking Risk Away From Layer 2 Interconnects

Documents

Transcript of 4. Data Center Networking Taking Risk Away From Layer 2 Interconnects