Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at...

56
Využití systémů řad Nexus 2000 a Nexus 5000 v návrhu přístupové v návrhu přístupové vrstvy sítě datového centra Jaromír Pilař ([email protected] ) Consulting Systems Engineer, CCIE 2910 © 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential Presentation_ID 1

Transcript of Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at...

Page 1: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Využití systémů řad Nexus 2000 a Nexus 5000 v návrhu přístupovév návrhu přístupové vrstvy sítě datového centra

Jaromír Pilař ([email protected])Consulting Systems Engineer, CCIE 2910

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Page 2: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Agenda

1. Nexus 5000 and Nexus 2000

2 Virtualized access switch2. Virtualized access switch

3. I/O consolidation in access layer

2

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 3: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

N 5000 d 2000Nexus 5000 and 2000

Platform descriptionat o desc pt o

3

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 4: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Nexus 5000 Systems Product PortfolioIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data CenterIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center

Nexus 5000SwitchFamily 56 Port L2 Switch28 Port L2 SwitchFamily 56-Port L2 Switch

• 40 Ports 10GE/FCoE/DCE, fixed• 2 Expansion Modules

28-Port L2 Switch• 20 Ports 10GE/FCoE/DCE, fixed• 1 Expansion Module

FC + Ethernet • 4 Ports 10GbE/FCoE/DCE

Fibre Channel • 8 Ports 1/2/4G FC

ExpansionModules Ethernet

• 6 Ports 10GE/FC E/DCE• 4 Ports 1/2/4G FC 10GE/FCoE/DCE

PartnersSW FCoE/DCE + 2x10GE2x10GE/DCE/FCoE 2x10GE

4

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

OS

Cisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OS

Mgmt

Page 5: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Nexus 2000 Fabric Extender1GE Connectivity1GE Connectivity

5

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 6: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

N 5000 d 2000Nexus 5000 and 2000

Virtualized access switchtua ed access s tc

6

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 7: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center ArchitectureEvolution of the Hierarchical Design

The Data Center Architecture is based on a hierarchical design model

Layer 3

Core Aggregation block contains

the access and aggregation layersC id l 3

Layer 2

Layer 3

Services

Aggregation Core provides layer 3

boundary to the rest of the network

Dedicated service switches Servicesprovide application load balancing, firewall, etc.

Architecture is based on optimized design for control plane stability

Access

design for control plane stability and scalability

Need to understand how the design needs to evolve to accommodate server applications

7

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

accommodate server, applications and facilities requirements

Page 8: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Datacenter network fabricTop-of-rack Modular

8

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 9: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureVirtualized Access Switch

Nexus 5010/5020 Dual N5K supported as of NX-OS 4.1(3)N1

Nexus 5000/2148T Virtualized Access Switch provides a number of design options to address evolving g p gData Center requirements

Fabric Extender provides for flexibility in the design of the physical topologiestopologies

Aids in building larger layer 2 designs safely Support of latest spanning tree pp p g

enhancements Single virtual access switch

(Simplifies the layer 2 design)S t f 16 10GE Support of 16-way 10GE Etherchannel combined with vPC in NX-OS 4.1(3)N1 provides for increased network capacity Nexus 2148T Fabric

4 x 10GE Fabric Links per Fabric Extender

9

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Extender– 48 GE Portsp

(CX-1 Cu)

Page 10: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center ArchitectureN5K/N2K - Logical Topology

1 Cisco Nexus 2148T Fabric1. Cisco Nexus 2148T Fabric Extender (N2K) and Nexus 5000 (N5K) Pod

2. N2K + N5K Pod t t ki

Each Virtualized Access Switch Pod configured to support

represents networking Access layer

3. Nexus 7000 at Distribution Layer

up to 576 1GE server ports at FCS

y

Nexus 5000/2000 Virtualized Access

Switch Pods . . .

10

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 11: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureDistributed Access Fabric

De-Coupling of the Layer 1 and Layer 2 Topologies

Optimization of both Layer 1 (Cabling) and Layer 2 p y ( g) y(Spanning Tree) Designs

Mixed cabling environment (optimized as required)

Flexible support for Future RequirementsFlexible support for Future Requirements

Nexus 5000/2000

. . .

11

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Combination of EoR and ToR cabling

Mixed ToR & EoR

Page 12: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Nexus 5000 and 2000

St ti i i P tStatic pinning, Port Channel and Virtual Port Channel on Nexus 5000C a e o e us 5000

12

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 13: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Fabric ExtenderF b i M d

Static PinningFabric Modes

Fabric Extender associates (pins) a server side (1GE) port with ana server side (1GE) port with an uplink (10GE) port

Server ports are either individually pinned to specific uplinks (static p p p (pinning) or all interfaces pinned to a single logical port channel

Behavior on FEX uplink failure depends on the configuration P t Ch l

Server Interface goes down

depends on the configuration Static Pinning – Server ports

pinned to the specific uplink are brought down with the failure of

Port Channel

brought down with the failure of the pinned uplink

Port Channel – Server traffic is shifted to remaining uplinks based

t h l h hServer Interface

stays active

13

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

on port channel hash stays active

Page 14: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Static Pinning

A B C D

N5k01

Fabric Ports1,2,3,4

N5k01

Fabric Ports1,2,3,4

N5k01

Fabric Ports1,2,3,4

N5k01

Fabric Ports1,2,3,4

1-48

N2k01Host Ports N2k01

1-24 25-48

Host Ports N2k01

1-16 17-32

Host Ports

33-48

N2k01

1-12 13-24

Host Ports

25-36 37-481-48

pinning max-links 1 pinning max-links 2

1-24 25-48

pinning max-links 3

1-16 17-32 33-48

pinning max-links 4

1-12 13-24 25-36 37-48

14

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 15: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Port-Channeling

1. With Static Pinning if a fabric uplink port fails, the associated HIFs are shut

Aassociated HIFs are shut down

2. With Port-Channeling if a f b i li k f il HIF

N5k01fabric uplink fails HIFs use the remaining fabric uplinks.

Fabric Ports

H t P t

1,2,3,4

1-48

N2k01

i i li k 1

Host Ports

pinning max-links 1

15

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 16: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureDual Supervisor – NXOS 4.0(1a)N2p ( )

Prior to the 4.1(3) release (Q3CY09) a Fabric Extender can ( )only be actively connected to a single N5K

During the link-up SDP exchange Fabric Extender identifies if it

N5K-A N5K-B

Fabric Extender identifies if it actively connected to an N5K

If the SDP identifies a different N5K as already being actively connected

SDP exchange as already being actively connected to the Fabric Extender the newly connected N5K port will err-disable

No prioritization between N5KErr-disable

Potential Race Condition if FEX connected to two N5K

Manual process can be used to

16

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

pprovide a backup configuration

Page 17: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureDual Supervisor – NX-OS 4.1(3)xxp ( )

NX-OS 4.1(3) provides vPC support on the N5K either for use ppto provide server NIC teaming or to provide dual supervisor configuration for the virtualized switch

In the redundant supervisor mode the Etherchannel fabric uplink is split across two N5Ks

Static pinning is not supported in a redundant supervisor mode

Server ports appear on both N5K Currently configuration for all ports

must be kept in sync manually on both N5Ks (N2K has in fact 2 configuration points in this setup)

17

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

configuration points in this setup)

Page 18: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Fabric ExtenderPort Channel Configurationg

interface port-channel1switchport mode fex-fabricdescription Fabric Extender 100fex associate 100

Configure the Physical Ports as

interface Ethernet1/1switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100

Physical Ports as Members of the

Fabric EtherChannel

interface Ethernet1/2switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100

Configure the Port interface Ethernet1/3switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100

interface Ethernet1/4

Configure the Port Channel and Its Members to be

Associated with a Specific Fabric

Extenderinterface Ethernet1/4switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100

fex 100

Extender

18

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

pinning max-links 1description Fabric Extender 100 – Using Etherchannel 1

Page 19: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Fabric ExtenderVirtual Port Channel Configurationg

interface port-channel1switchport mode trunkswitchport trunk allowed vlan 1,10vpc peer-link

Configure the vPC Peer Link (Full vPC Configuration not included in this

example)interface port-channel100

switchport mode fex-fabricvpc 100fex associate 100

interface Ethernet1/17

example)

interface Ethernet1/17switchport mode trunkswitchport trunk allowed vlan 1,10channel-group 1 mode active

interface Ethernet1/18switchport mode trunk

Configure the Physical Ports as Members of the

Fabric EtherChannelswitchport trunk allowed vlan 1,10channel-group 1 mode active

interface Ethernet1/37switchport mode fex-fabricchannel-group 100fex associate 100fex associate 100

interface Ethernet1/38switchport mode fex-fabricchannel-group 100fex associate 100

Configure the Port Channel and Its Members to be

Associated with a Specific Fabric

19

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

fex 100pinning max-links 1

Extender

Page 20: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitecturevPC Redundancy Models Dual ChassisvPC Redundancy Models – Dual Chassis vPC provides two redundancy designs for the virtualized access switch Option 1 - MCEC connectivity from the serverp y

Two virtualized access switches bundled into a vPC pair Full redundancy for supervisor, line card, cable or NIC failure Logically a similar HA model to that currently provided by VSS Logically a similar HA model to that currently provided by VSS

vPC peersTwo Virtualized access switches Each with a Single Supervisor

20

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

MCEC from server to the access switch

Page 21: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitecturevPC Redundancy Models Dual SupervisorvPC Redundancy Models – Dual Supervisor vPC Option 2 – Fabric Extender connected to two Nexus 5000

From the server perspective a single access switch with each line card p p gsupported by redundant supervisors

Full redundancy for supervisor, fabric via vPC and cable or NIC failure via active/standby NIC redundancyL i ll i il HA d l t th t tl id d b d l Logically a similar HA model to that currently provided by dual supervisor based modular switch

Fabric Extender dual homed to redundant Nexus 5000

21

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 22: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureVirtual Port Channel Design ConsiderationsVirtual Port Channel – Design Considerations

Dual tiered vPC is not supported with the NX-OS 4.1(3) release

4. TCP Flow also from SRC A appears

on fabric interface Po 3. TCP Flow from SRC A

appears on fabric vPC provides a logical port

channel interface on the N5K In the unsupported configuration

shown each N2K is attached with a

100ppinterface Po 200

shown each N2K is attached with a port channel fabric interface and then carried over that interface is a second server port channel interface

5. MAC flap between two

different fabric ports

Two dependent layers of Etherchannel hashing (Server and N2K) for the same flowsT i f PC h ill

2. N2K hashes both flows to the same

ports

Two tiers of vPC as shown will result in MAC addresses flapping between two fabric ports

Timeframes to support this

1. Server hashes two flows up

alternate uplinks

N5K

22

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Dual Tier vPC shown above is notsupported with the 4.1(3) N5K release

ppconfiguration is still TBD

Page 23: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureVirtual Port Channel Design ConsiderationsVirtual Port Channel – Design Considerations

Two independent vPC configurations are supported onconfigurations are supported on the same N5K

vPC upstream to the N7K is independent of the downstreamindependent of the downstream vPC connecting the N2K

Independent hashing decisions for the upstream and d t t ffi fldownstream traffic flows

23

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 24: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureVirtual Port Channel – Design Considerationsg

Multiple Interfaces on the Single Interface

VMVMVM

Interfaces on the same N5K

requires HW resource

VMVMVM

gon each N5K

VM #4

VM #3

VM #2

VM #4

VM #3

VM #2

Nexus 5000 is limited to 12 physical (HW) Ethernet port channels (increasing to 16 in NX-OS 4 1(3))(increasing to 16 in NX OS 4.1(3))

A MCEC vPC configuration with only two links (one per N5K) does ‘not’ consume a HW port channel resource

A HW port channel will be required in a vPC environment only if there is

24

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

A HW port channel will be required in a vPC environment only if there is more than one interface on the same N5K (e.g. 4 x 10GE links)

Page 25: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureVirtual Port Channel – FCoE Considerations

SAN ‘B’SAN ‘A’vPC - MCEC

Two logical views of the

VM VM VM VMVMVM

same physical topology

#4#3#2VM #4

VM #3

VM #2

In a Unified I/O configuration (FCoE) we have two distinct topologies

Isolated access switches - SAN ‘A’ and SAN ‘B’

Combined access switches – vPC supporting MCEC

To ensure correct forwarding behaviour ‘vfc’ interface can only be

25

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

g yassociated with a vPC etherchannel (only one physical interface per switch)

Page 26: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureNexus 5000 & 2000 Deployment Optionsp y p

Not SupportedNot SupportedNot SupportedNot Supported

With PC

26

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

With vPCWithout vPC

FCS OptionsFCS OptionsFCS OptionsFCS Options vPC vPC -- Q3CY09Q3CY09vPC vPC -- Q3CY09Q3CY09 TBDTBDTBDTBD

Page 27: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Nexus 5000 and 2000

Fabric Extender Layer 2 Design Considerationses g Co s de at o s

27

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 28: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureN2K/N5K Spanning Tree Design Considerations

Root BridgeHSRP A ti

Secondary Root B idHSRP Active Bridge

HSRP Standby

BPDU Guard Global BPDU Filter reduces the spanning

tree load (BPDUs generated on a Host

Bridge Assurance

UDLD

gPort)

VMWare Server Trunk Needs to Carry

VSwitch

yMultiple VLANs which can increase the STP

load

28

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

VM #1

VM #4

VM #3

VM #2

Page 29: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureN2K/N5K Spanning Tree Design Considerations

Nexus-5000# show spanning-tree interface ethernet 100/1/48 detail

Port 560 (Ethernet100/1/48) of VLAN0100 is designated forwarding Port path cost 4 Port priority 128 Port Identifier 128 560Port path cost 4, Port priority 128, Port Identifier 128.560Designated root has priority 24776, address 0023.ac64.73c3Designated bridge has priority 32968, address 000d.eca4.533cDesignated port id is 128.560, designated path cost 2Timers: message age 0, forward delay 0, hold 0Number of transitions to forwarding state: 1The port type is edgeLink type is point-to-point by default BPDU Guard Is Enabled by Default and Link type is point to point by defaultBpdu guard is enabledBPDU: sent 215784, received 0

BPDU Guard Is Enabled by Default and Can Not be Disabled on FEX Server Ports

interface port-channel200switchport mode trunkswitchport mode trunkswitchport trunk allowed vlan 200-202spanning-tree port type network

interface Ethernet1/33switchport mode trunkswitchport trunk allowed vlan 200-202udld enable

Bridge Assurance Requires the Port Type to be

Configured as ‘network’

udld enablechannel-group 200 mode active

interface Ethernet1/37switchport mode trunkswitchport trunk allowed vlan 200-202udld enablechannel-group 200 mode active

29

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

channel group 200 mode active

Nexus5000(config)# spanning-tree port type edge bpdufilter default Global BPDU Filter

Page 30: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitecturevPC - Spanning Tree Design Considerations

Both vPC PeersAct as the

Enabling vPC on the access to aggregation links improves

Single Logical Link to STP

Act as the default GW

aggregation links improves layer 2 scalability Removing physical loops

out of the layer 2 topologyvPC

Link to STPy p gy Reducing the STP state on

the access and aggregation layer

Fabric Links(No

Spanning Tree)

The use of vPC does result in a reduction of logical port count on the aggregation but does involve CFS synchronization of

Server PortsBPDU Guard

involve CFS synchronization of state between the two aggregation nodes

30

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 31: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureSupported Topologies and Design Considerations

Fabric Extender design needs to take into account topology

Spanning Tree - Rapid

PVST+ or to take into account topology and hardware capabilities of the current generation FEX

Currently all Fabric Extender

PVST+ or MST

Switch server ports are hard coded as STP edge ports BPDU Guard is enabled and

can not be disabled

Switch Fabric No STP

Spanning Tree Edge Ports

can not be disabled

Current generation HW does not support port channel between any two server ports

th N 5000on the same Nexus 5000 Currently all Fabric Extender

server ports support 1000BaseT only

VSwitch

VM #1

VM #4

VM #3

VM #2

31

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

1000BaseT only #1 #4#3#2

Page 32: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitecturePort Channel Design Considerations – 4.0(1a)N2

With NX-OS 4.0(1a)N2 the Nexus Fabric Extender does not support

S tpp

Port Channel (Ether Channel) between any two server ports (Logical Interfaces) on the same Nexus 5000 N5K

Server ports

Not possible to bundle two or more ports on the same Fabric Extender

Not possible to bundle ports spreadNot possible to bundle ports spread across two or more Fabric Extenders that are associated with the same N5K (line cards in the same virtual switch)same virtual switch)

Servers are unable to use a port channel configuration unless vPC is enabled (two port vPC only)

32

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 33: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitecturePort Channel Design Considerations – 4.1(3)N1

With the Cronulla release (Q3CY09) Virtual Port Channel (vPC) can be N5KA N5KB( )used to create a server side port channel between two ports on different N5K’s

Not possible to bundle two or more

N5KA N5KB

Not possible to bundle two or more physical links to two Logical Interface Ports even in a vPC bundle

Not possible to bundle ports spreadNot possible to bundle ports spread across two or more Fabric Extenders that are associated with the same N5K (line cards in the same virtual switch)switch)

Servers are unable to use a port channel configuration unless vPC is enabled

Maximum of two ports supported in a vPC bundle

33

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 34: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureDownstream Device Design Considerations

Fabric Extender server ports are configured as either STP edge

Attached devices canutilize trunks and

source multiple MAC addressesg g

access or edge trunk ports BPDU Guard is enabled by default

and can not be disabled Fabric Extender ports can be

configured as trunks and can receive multiple MAC addressesI ll F b i E t d d i th In all Fabric Extender designs the layer 2 topology must not require STP to be configured on the attached device

Fabric Extender ports are STP edge ports, BPDU Guard is

enabled. Downstream devices

VSwitch

VM #1

VM #4

VM #3

VM #2

STP Disabled

34

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

enabled. Downstream devices can not generate BPDU’s.

STP Disabled

Page 35: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureFlexlink and Fabric Extender

Design Topologies that do not require STP for downstream link qrecovery can be configured

Flexlink is one option for connecting a downstream switch that is STP independentindependent

Flexlink disables Spanning Tree on the active and backup interfacesBPDU d d i HW

STP edge BPDUs are dropped in HW on

ingress and not transmitted upstream on the ‘flexlink’ ports

Note: Extending layer 2 downstream

gports

Note: Extending layer 2 downstream of the N2K is not a recommended best practices solution Flexlink

configured on the stub switch

li k

35

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

dc11-2960-1#sh spanning-tree interface gig 0/25 detail no spanning tree info available for GigabitEthernet0/25

uplinks

Page 36: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Data Center Access ArchitectureFlexlink and Fabric Extender

Spanning Tree should ‘not’ be disabled on either the Nexus 5000 nor the downstream switches (e.g. Cisco 2960)

External devices can still createExternal devices can still create loops in the topology

DO NOT extend the downstream device topologydevice topology

This design suitable only for migration environments

2960 for 10/100 iLO

1GE Blade Switches

36

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

BPDU Guard, port security, storm control, 802.1x

Page 37: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Nexus 5000 and Nexus 2148 Fabric Extender

Fabric Extender Configurations

Nexus 5000

DC Access Layer consolidation

37

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 38: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Datacenter network fabricTop-of-rack Modular

38

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 39: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Datacenter storage fabricgTop-of-rack Modular

Smaller switch per bi t

Modular switch for group of cabinets

cabinet

Pair of switches for dual-homing

Uplinks to redundant core layer - 2/4/10G FC

g p

Pair of switches for dual-homing

Uplinks to redundant core layer

2/4/10G FCcore layer 2/4/10G FC

Use of portchanneling

Larger port count in core is required

- 2/4/10G FC

Use of portchanneling

Lower port count in core is required

C

Pro: Efficient CablingPro: Improved CoolingCon: Number of Devices/MgmtCon: Overhead protocol load

Core

Pro: Number of Devices/MgmtPro: Overhead protocol loadCon: Cabling ChallengesCon: Cooling Challenges

Core

Edge

Core

Edge

39

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 40: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Where is the main consolidation potential ?Where is the main consolidation potential ?1. Majority of ports in fabric is in access layer regardless

of fabric type => access layer has the highest potential for consolidation

2. Question is which technology can serve different d f diff t f b i d h it t d ?needs of different fabrics, do we have it today?

3. What are the requirements and what are other driving factors?factors?

40

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 41: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Consolidation - historyConsolidation history1. Previous attempts

Fibre ChannelNever credible as data network infrastructre

InfiniBandInfiniBandNot Ethernet

iSCSINot Fibre Channel

2. Key enabling technologies: PCI-Express and 10GEBefore PCI-Express there was not enough I/O bandwidth in the

servers

It needs to be Ethernet but

41

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

It needs to be Ethernet, but…1 GE didn’t have enough bandwidth

Page 42: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Drivers for 10GE to the Servers

Multicore CPU Architectures Allowing Bigger and M lti l W kl d th S M hi

Drivers for 10GE to the Servers

Multiple Workloads on the Same Machine

S Vi t li ti D i i th N d f MServer Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server

Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections

42

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Demand for Higher Bandwidth Network Connections

Page 43: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

I/O Consolidation in the Network

Processor

Memory

Processor

MemoryMemory Memory

I/O SubsystemI/O I/O I/OLA

N

orag

e

IPC

y

AN

orag

ePCL

Sto I LA

Sto IP

43

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

IPC: Inter-Process Communication

Page 44: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

I/O Consolidation in the Host

1. Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs

2. Limited number of interfaces for Blade Servers

FC TrafficFC HBA

FC TrafficFC HBAAll Traffic Goes over

10 GE

CNA

CNA

NIC Enet Traffic

NIC Enet Traffic

NIC Enet Traffic

HCA IPC Traffic

44

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

IPC TrafficHCA

Page 45: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Cabling and I/O Consolidation

45

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 46: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Unified I/O Use Case

Today:

FC HBA

FC HBA

FC Traffic

FC Traffic

FC HBA

FC HBA

FC HBA

FC HBA

1. Parallel LAN/SAN Infrastructure

2. Inefficient use of Network Infrastructure

3 5 ti hi hFC HBA

NIC

FC Traffic

Enet Traffic

FC HBA

NIC

FC HBA

NIC

3. 5+ connections per server – higher adapter and cabling costs

Adds downstream port costs; cap-ex and op-ex

NIC Enet TrafficNICNIC

p p

Each connection adds additional points of failure in the fabric

Power and cooling

4. Longer lead time for server provisioning

5. Multiple fault domains – complex di ti

46

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

diagnostics

6. Management complexity –firmware, driver-patching, versioning

Page 47: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

What is Data Center Bridging?

Provides class of service flow control. Ability to support Priority-based Flow

BenefitFeature

g g

Grouping classes of traffic into “Service Lanes”IEEE 802.1Qaz, CoS based Enhanced Transmission

CoS Based BW Management

o des c ass o se ce o co o b y o suppostorage traffic (IEEE 802.1Qbb)

o ty based oControl (PFC)

End to End Congestion Management for L2 network (IEEE 802.1Qau)

Congestion Notification (BCN/QCN)

Eliminate Spanning Tree for L2 topologies L2 Multi-path for Unicast &

Auto-negotiation for Enhanced Ethernet capabilities DCBX (Switch to NIC)

Data Center Bridging Exchange

Provides ability to transport various traffic types (e.g. Storage, RDMA)

Lossless Service

p g p g

Utilize full Bi-Sectional bandwidth with ECMPL2 Multi path for Unicast & Multicast

47

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Page 48: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Merging the requirements

1. LAN/IP1. LAN/IPMust be Ethernet

Losing some frames is good for TCP

2. StorageMust follow the Fibre Channel model

Losing frames is not an option

3. IPCDon’t care of the underlying network, provided that

It is cheap

48

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

It is low latency

It supports APIs like OFED, RDS, MPI, etc. . .

Page 49: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

What is Fibre Channel over Ethernet?

1. From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… an Ethernet

cloud

2. From an Ethernet standpoints it’s2. From an Ethernet standpoints it sYet another ULP (Upper Layer Protocol) to be transported,

but… a challenging one!

3. And technically…

FCoE is an extension of Fibre Channelf

49

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

onto a Lossless Ethernet fabric

Page 50: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

FCoE Enablers and Encapsulation

1. 10Gbps Ethernet

2. Lossless EthernetMatches the lossless behavior guaranteed in FC by B2B credits

3. Ethernet jumbo framesMax FC frame payload = 2112 bytes

T t l f i 2180 b t

Same as a physical FC frame

Normal ethernet frame, ethertype = FCoE

Total max frame size = 2180 bytes

Ethe

rnet

Hea

der

FCoE

Hea

der

FCH

eade

r

FC Payload CR

C

EOF

FCS

Same as a physical FC frame

50

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

E

Control information: version, ordered sets (SOF, EOF)

Page 51: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Unified I/O Use Case

Today:

Management

SAN BSAN ALAN

1. Aggregation/Core it h

FC HBA

FC HBA switchesFC HBA

NIC

2. Access – Top of the Rack switches

NIC

Rack switches

3. Servers

51

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

EthernetFC

Page 52: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

FCoE I/O Consolidation BenefitFewer HBA/NIC’s per Server

FC E CNA

FC HBA

FC HBA

FC Traffic

FC Traffic FC EFCoE CNA

FCoE CNA

FC HBA

NIC

FC Traffic

Enet TrafficFCoE

&Enet@

NIC Enet Traffic

@10GE

FCoE Converged Network Adaptor

52

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Customers purchase fewer NIC’s and HBA’s

Page 53: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Unified I/O Use Case

TodayUnified I/O Phase 1Unified I/O Phase 11. Reduction of server

d t

Management

SAN BSAN ALAN adapters2. Fewer Cables3. Simplification of access

l & blilayer & cabling4. Gateway free

implementation - fits in installed base of existing

FCoE Switch

installed base of existing LAN and SAN

5. L2 Multipathing Access –Distribution

6. Lower TCO7. Investment Protection

(LANs and SANs)

53

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

FCoEEthernetFC

8. Consistent Operational Model

9. One set of ToR Switches

Page 54: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Unified I/O Use Case

Unified I/O Phase 21 Elimination of parallel network

Unified I/O Phase 2

Enh E Fabric w/ FCoE

1. Elimination of parallel network infrastructure

2. L2/L3 Multipathing end to endManagement

StorageArrays

3. Faster infrastructure provisioning

4. Lower TCO5. Disk array access via

Enhanced Ethernet or Native FC

FCoE Switch

54

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

Enh E and FCoEFC

Page 55: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

Unified I/O Use Case

Unified I/O Phase 31 Datacenter wide Unified

Today:Unified I/O Phase 1 (Mid 2008)Unified I/O Phase 31. Datacenter wide Unified

Fabric for LAN and SAN2. L2/L3 Multipathing end to

endManagement

SAN BSAN ALANManagement

SAN BSAN ALANEnh E Fabric w/ FCoE

StorageArrays

end3. Consistent network policies

across datacenter4 Lo er TCO

Management

4. Lower TCO

55

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID

DCE and FCoEEthernetFC

DCE and FCoEEthernetFCEnh E and FCoE

Page 56: Využití systémů řadpalo/Rozne/cisco-expo... · Nexus 2000 a Nexus 5000 ... Nexus 7000 at Distribution Layer up to 576 1GE server ports at FCS Nexus 5000/2000 Virtualized Access

56

© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID