Post on 08-Mar-2018
Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture
BRKARC-3452
Nicolas Delecroix
Technical Marketing Engineer
Data Center Group – Nexus Business Unit
ndelecro@cisco.com
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5000 Evolution
Customers
Nexus 5010 28-Port, 1RU
Nexus 5020 56-Port, 2RU
Nexus 5548 48-Port, 1RU
Nexus 5596 96-Port, 2RU
Nexus 5596T 10Gbase-T, 2RU
3
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Customers
Nexus 5010 28-Port, 1RU
Nexus 5020 56-Port, 2RU
Nexus 5548 48-Port, 1RU
Nexus 5596 96-Port, 2RU
Nexus 5596T 10Gbase-T, 2RU
Nexus 6004
96 port 40G, 4RU
Introducing Cisco Nexus 5600/6000
4
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Customers
Nexus 5672
48 port 10G, 6 port 40G, 1RU
Nexus 6004
96 port 40G, 4RU
Nexus 56128
96 port 10G, 8 port 40G, 2RU
Nexus 5010 28-Port, 1RU
Nexus 5020 56-Port, 2RU
Nexus 5548 48-Port, 1RU
Nexus 5596 96-Port, 2RU
Nexus 5596T 10Gbase-T, 2RU
5
Introducing Cisco Nexus 5600/6000
Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture
BRKARC-3452
Nicolas Delecroix
Technical Marketing Engineer
Data Center Group – Nexus Business Unit
ndelecro@cisco.com
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
7
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
6 fixed 40G QSFP interfaces
Nexus 5672 Chassis
• All ports provide Ethernet and FCoE
• 16 Unified Ports (orange) also provide 2/4/8G FC
• L2 and L3 VXLAN
Port-Side View
8
48 fixed 1/10G SFP+ interfaces
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5672 Chassis Power Supply and Fans
9
Power Supply 1+1
Redundancy
Console Mgmt0 USB Fan Module 2+1
Redundancy
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 56128 Chassis
• 48 Fixed 10G SFP+ Ports, 4 fixed 40G QSFP+ ports
• 2 Expansion Slots
• Expansion module provides 24 Unified ports 10G Ethernet/FCoE or 2/4/8G FC, and 2 ports 40G Ethernet/FCoE
Port-Side View
10
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 56128 Chassis Power Supply and Fans
11
Power Supply 2+2
Redundancy
Console Mgmt0 USB Fan Module 3+1
Redundancy
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 6004 Chassis Port-Side View
12
4RU
12 QSFP+ ports
Expansion Module
N6K-6004-M12Q
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 6004 Chassis
• Six power supply slots; a minimum of three is required. Supports both 3 + 1 and 3 + 3 redundancy.
• Each power supply is rated 1100W, 90—240 VAC.
• The chassis has four fan trays. A minimum of three is required.
Power Supply and Fans
13
Power Supply 3 + 3
Grid Redundancy
or 3 + 1 Redundancy
Console Mgmt0 USB
Fan Module 3+1
Redundancy
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Unified Port Line-Card Expansion Module - N6004X-M20UP
For Nexus 6004
14
• 20 * 10G SFP+ unified ports expansion module
• 2/4/8G FC port
• 1G/10G and FCoE
• Max number of native FC ports per chassis is 48 (as of NXOS 7.0(2)N1(1) release)
• SFP+ port allows support for a larger variety of optical transceivers
• Same transceivers supported as on 5600 10G ports – including 10G long distance, 1G copper and optical
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Airflow
• The units have a “port side” and a “fan side”. No front or back
• The units support both port side exhaust and port side intake – Port side exhaust (used for in server rack airflow alignment)
– Port side intake (used for network rack airflow alignment)
• Different power supply and fan modules are required for different airflow directions
15
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public 16
Feature Nexus 5600/6000 Nexus 5500
Native 40GE Support Yes 4xQSFP module – 10G flows
1GE Support Yes Yes
Unified Ports Yes Yes
FEX Scaling 48 (6004 L2), 24 (6004 L3, 5600) 24 with L2, 16 with L3
MAC Table 256K (shared with ARP/ND) 32K
IPv4 Unicast Routes 32K 16K
IPv4 ARP/Host Route Entries 128K 16K (shared with IPv6 ND and
mroute)
IPv4 Multicast Routes 32K 8K (shared with ARP)
N5600/6000 and N5500 Comparison
Latency ~1us 1.8us
L3 Throughput Line rate 160Gbps
L2 Throughput Line rate Line rate
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public 17
Feature Nexus 5600/6000 Nexus 5500
VLAN 4K 4K
VRF 4K 1K
Segment ID Yes No
ACL 4K 4K
L4ops for ACL 24 total (16 for UDP/TCP, 8 for TCP flag) 8 for UDP and 8 for TCP
Packet Buffer 25MB per 3xQSFP (or 12xSFP) ports
2.1MB per 10G port
640KB per 10G port
vPC, FabricPath Yes Yes
N5600/6000 and N5500 Comparison (cont.)
IPv6 Multicast Routes 32K 8K (shared with ARP, ND. Limit is
with /64 routes)
IPv6 ND Entry/Host Route Entries 85K (shared with MAC and ARP) 8K (shared with mroute and ARP)
IPv6 Unicast Routes 8K (shared with IPv4) 8K (shared with IPv4)
Dynamic Fabric Automation Yes No
VXLAN Yes* No
*Software roadmap feature
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
18
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 2000 100M/1G Fabric Extender Portfolio
19
Nexus 2224TP Nexus 2248TP Nexus 2248TP-E
Uplink Ports 2 x 10G SFP+ 4 x 10G SFP+ 4 x 10G SFP+
Host Facing Ports 24 x 100/1000BASE-T RJ45
48 x 100/1000BASE-T RJ45
48 x 100/1000BASE-T RJ45
Oversubscription 1 : 1.2 1 : 1.2 1 : 1.2
FCoE N/A N/A N/A
Buffer 3.8MB 7.6MB 32MB
Shared Buffer No No Yes
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 2000 1G/10G Fabric Extender Portfolio
20
Nexus 2232PP Nexus 2232TM Nexus 2232TM-E Nexus 2248PQ
Uplink Ports 8 x 10G SFP+ 8 x 10G SFP+ 8 x 10G SFP+ 4 x 40G QSFP+
Host Facing Ports 32 x SFP/SFP+ (1/10G)
32 x 1/10GBASE-T RJ45
32 x 1/10GBASE-T RJ45
48 x SFP/SFP+ (1/10G)
Oversubscription 1 : 4 1 : 4 1 : 4 1 : 3
FCoE Yes No Yes (30m / Cat6a/7) Yes
Buffer 10.2MB 10.2MB 10.2MB 32MB
Shared Buffer No No No Yes
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 2000 Fabric Extender Scalability
• Nexus 5600 and 6000 support all types of FEX, including HP, Fujitsu, Dell, IBM FEX for blade servers. Adapter-FEX and VM-FEX are also supported.
21
+ 48
xF
EX
Cisco Nexus 5600/6000 Cisco Nexus 2000 FEX
Nexus
5000
Nexus 5500 Nexus 5600 Nexus
6004
Nexus
7000
Nexus 7700
L2 12 24 24 48 48 64
L3 x 16 24 24 48 64
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
22
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5672 - 72 x 10G Unicast Throughput
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 2544 Unicast L2 Port Pair 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 2544 Unicast L3 Port Pair 100 100 100 100 100 100 100 100 100 100 100 100 100
0
10
20
30
40
50
60
70
80
90
100
Throughput (% of line rate)
Packet Size (Bytes)
23
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5672 - 72 x 10G Multicast Throughput
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 3918 Mcast L2 One to Many Accumulated 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L3 One to Many Accumulated 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L2 One to Many Distributed 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L3 One to Many Distributed 100 100 100 100 100 100 100 100 100 100 100 100 100
0
10
20
30
40
50
60
70
80
90
100
Throughput (% of line rate)
Packet Size (Bytes)
24
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5672 – 72 x 10G Unicast 100% Load Latency
25
64 128 256 512 1024 1280 1518 2240 4096 9216
RFC 2544 unicast L2 port pair 1001 1021 1091 1111 1105 1104 1104 1103 1101 1098
RFC 2544 unicast L3 port pair 937 963 1029 1048 1047 1046 1045 1044 1043 1040
0
200
400
600
800
1000
1200
1400
1600
Average Latency (Nano Seconds)
Packet Size (Bytes)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 Performance – Fully Loaded Switch
26
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 Performance – Spirent Traffic Generator
27
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 - 96 x 40G Unicast Throughput
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 2544 Unicast L2 Port Pair 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 2544 Unicast L3 Port Pair 100 100 100 100 100 100 100 100 100 100 100 100 100
0
10
20
30
40
50
60
70
80
90
100
Throughput (% of line rate)
Packet Size (Bytes)
Spirent third party performance report for Nexus 6004
28
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 - 96 x 40G Multicast Throughput
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 3918 Mcast L2 One to Many Accumulated 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L3 One to Many Accumulated 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L2 One to Many Distributed 100 100 100 100 100 100 100 100 100 100 100 100 100
RFC 3918 Mcast L3 One to Many Distributed 100 100 100 100 100 100 100 100 100 100 100 100 100
0
10
20
30
40
50
60
70
80
90
100
Throughput (% of line rate)
Packet Size (Bytes)
Spirent third party performance report for Nexus 6004
29
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 - 384 x 10G Unicast 100% Load Latency
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 2544 Unicast L2Port Pair
1.028 0.99 1.005 1.023 1.121 1.118 1.111 1.116 1.11 1.107 1.11 1.108 1.101
RFC 2544 Unicast L3Port Pair
1.032 1.001 1.011 1.03 1.122 1.121 1.116 1.118 1.113 1.111 1.112 1.11 1.106
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Average Latency (microseconds)
Packet Size (Bytes)
30
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 - 384 x 10G Multicast 100% Load Latency
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
RFC 3918 Mcast L2 One-to-ManyAccumulated
1.004 0.989 1.029 1.017 1.095 1.086 1.079 1.084 1.083 1.082 1.082 1.08 1.077
RFC 3918 Mcast L3 One-to-ManyAccumulated
1.067 1.051 1.061 1.064 1.087 1.083 1.079 1.081 1.08 1.079 1.078 1.076 1.073
RFC 3918 Mcast L2 One-to-ManyDistributed
0.951 0.949 0.954 0.962 1.019 1.042 1.043 1.042 1.042 1.042 1.043 1.043 1.043
RFC 3918 Mcast L2 One-to-ManyDistributed
0.96 0.958 0.963 0.974 1.019 1.044 1.044 1.044 1.044 1.044 1.044 1.044 1.044
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Average Latency (microseconds)
31
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004 - 384 x 10G Unicast and Multicast 100% Load Jitter
64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100
Unicast L2 Jitter (ns) 8 5 8 5 6 8 6 7 6 10 10 9 9
Unicast L3 Jitter (ns) 8 5 9 5 6 8 6 7 6 10 10 9 10
Multicast L2 Jitter (ns) 9 6 9 5 5 8 6 7 6 10 9 9 9
Multicast L3 Jitter (ns) 9 5 9 5 5 8 6 7 6 9 9 9 9
0
2
4
6
8
10
12
Average Jitter (nanoseconds)
Packet Size (Bytes)
32
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
33
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
UPC 1
48xSFP+
Unified Fabric Controller (UFC)
Switch Fabric
6xQSFP+ CPU
Supervisor
UPC 0
Cisco Nexus 5672 Internal Architecture
34
UPC 2 UPC 3 UPC 4 UPC 5 UPC 6
FC PHY
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
UPC 1
48xSFP+
Unified Fabric Controller (UFC)
2x
QSFP+
CPU
Supervisor
UPC 2 UPC 3 UPC 4 UPC 5 UPC 6
2x
QSFP+
UPC 7 UPC 8 UPC 9 UPC 10 UPC 11 UPC 12
24xSFP+ 2x
QSFP+ 24xSFP+
2x
QSFP+
Slot 1 Slot 2
Cisco Nexus 56128 Internal Architecture
35
FC PHY FC PHY FC PHY FC PHY
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
UPC 1
UFC 1 UFC 2 UFC 3 UFC 4
Switch Fabric
CPU
Supervisor
UPC 0
Cisco Nexus 6004 Internal Architecture
36
UPC 2 UPC 3 UPC 4
12xQSFP+ 20xSFP+
FC PHY
UPC 1 UPC 2
12x40G LEM (M12Q) 20x10G LEM (M20UP)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Unified Port Controller - UPC ASIC • Multimode MAC; built-in PHY for 1, 10, and 40GE
• Packet parsing and rewriting
• Lookup engine and access control: L2, L3, FabricPath, ACL, FCoE, and policing
• Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast)
37
UPC
UPC
224Gbps
224Gbps
448Gbps
448Gbps
Twelve 10 GE
Three 40 GE
Front Panel Ports Fabric Connections
• Extra fabric bandwidth for SPAN and multi-destination traffic
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Packet Processing Flow Ingress Pipeline
38
Un
ified
Cro
ss
ba
r
Fa
bric
MAC
Unicast VOQ
(8 per egress port)
8000 Multicast VOQs
Ingress UPC
1, 10, and 40 GE
Buffer Allocation
Buffer Accounting
PAUSE Signal
Buffer
Management
Buffer
Packet
Rewrite
Ingress
SPAN
Replication
Parser
Ingress ACL
Ingress Policing
Forwarding
Lookup
VLAN Table
MAC Table
ARP/ND Table
IPv4/IPv6 Routing Table
FabricPath
FC routing Table
Security ACL
QoS ACL
SPAN Filter
Policing/Stats
Storm Control
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Packet Processing Flow Egress Pipeline
39
Un
ifie
d C
ros
sb
ar
Fa
bri
c
MAC
Egress UPC
Unicast buffer
Multicast
congestion
management
Unicast Queue
Buffer
Editing
TX SPAN
Replication
To Ingress
Buffer Management
Multicast Queue
Multicast
Replication
Buffer
Management
*Egress policing is not supported with current software
Scheduling Egress ACL
Egress Policing*
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Key Forwarding Tables
40
Host table: 256,000 -entry hashing table;
actual capacity is slightly less than 256,000
Host table: Shared between MAC, ARP, and
ND and /32 host route
Host table default carving: 128,000 MAC,
128,000 IP hosts
LPM table: 32,000 entries. Also known as
summary routes
IP Host Region (ARP/ND/Host
route/(*,G))
MAC Region Summary Routes (S,G)
Host Table* LPM Table(32,000) Mroute Table(64,000)*
25
6,0
00
En
trie
s
* Hardware table size. Please check configuration limit for software scaling
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Host Table Carving
• In Layer 2, virtualized environments, it can be interesting to have more than 128k MAC addresses on the switch
• MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region.
IP Host Region (ARP & ND & /32 Host route)
MAC Region
Host Table 2
56
,00
0 E
ntr
ies
41
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Carving Profiles
• Use of pre-defined carving profiles.
• HRT = Host Route Table = IP host region.
• STM = Station Table Management = MAC region.
Template Profile Description
hrt-128-stm-128 HRT size: 128k, STM size: 128k (default profile)
hrt-96-stm-160 HRT size: 96k, STM size: 160k
hrt-64-stm-192 HRT size: 64k, STM size: 192k
hrt-32-stm-224 HRT size: 32k, STM size: 224k
42
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 MAC/ARP Resource Carving CLI
• Specify the resource template to use: • switch(config)# hardware profile route resource service-
template template-name
• Save the config and reload the switch
• Show commands: • show hardware profile route resource template
• show hardware profile route resource template default
• show running-config hardware profile route resource template
• show startup-config hardware profile route resource template
43
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
44
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
QSFP 40G Bidirectional Transceiver
40G deployment with 10G cabling
45
• QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing Multi-mode fiber infrastructure
– Using the existing 10Gbit/s Multi-mode infrastructure
– QSFP-40G-SR-BD has dual LC Connector
– 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
QSFP-40G-SR-BD
46
40G-BD 40G-BD
40G-BD
10G-SR
10G-SR
10G-SR
10G-SR
Multimode
LC Duplex
Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF
LC duplex connectors
Wavelength: ~850nm and ~900nm
Pull-Tab Color: grey
Optical
DeMux
Multimode
LC Duplex
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
QSFP-40G-SR4
47
40G-SR4 40G-SR4
40G-SR4
10G-SR
10G-SR
10G-SR
10G-SR
MPO-12 MPO-12
MPO-12 LC Breakout
• Support 100M with OM3 MMF and 150M with OM4 MMF
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
QSFP-40G-CSR4
48
• Supports 300M with OM3 MMF and 400M with OM4 MMF
• Compatible with Cisco QSFP-40G-SR4
• Compatible with 10G SFP-SR
40G-CSR4 40G-SR4/CSR4
40G-CSR4
10G-SR
10G-SR
10G-SR
10G-SR
MPO-12 MPO-12
MPO-12 LC Breakout
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Multimode Ribbon Fiber 40GE
49
For QSFP-40G-SR4 and
QSFP-40G-CSR4
12-Fiber MPO Connector
• MPO/MTP connector with 12
pins
• Use 4 fiber pairs
− 4 TX and 4 RX allows for 40GE
− 4 unused fibers in the center
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
QSFP-40GE-LR4
50
Host
Card
IEEE Standard 40GBASE-LR4
4x ROSA
4x TOSA
LC TIA
Multiplexer
LC
• QSFP-40GE-LR4 can only be connected to each other. 10KM with Single mode fiber.
Can’t connect to 10Gbase-LR
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
FET-40G
• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 • Supported on Cisco Nexus 5600/6004 and Nexus 2248PQ-10G • Interoperable with FET-10G • Support for 100m distance with OM3
51
Cisco Nexus 5600/6000 Cisco Nexus 5600/6000
Cisco Nexus 2232PP and 2232TM-E 2232TM
Cisco Nexus 2248TP-E 2248TP Cisco Nexus 2248PQ-10G
FET-40G
FET-40G
FET-10G
FET-40G
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
52
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
ACL Types and Features
• Security ACL
− MAC, IPv4, and IPv6 ACLs
− PACL: ACL enabled under L2 interface
− VACL: ACL enabled for L2 VLAN traffic
− RACL: ACL enabled for routed traffic
− RBACL: Role-based ACL for CTS (roadmap feature)
• ACL for QoS classification and Policing
• Policy Based Routing (PBR)
– User-configured ACL to redirect traffic
• ACL for SPAN/ERSPAN
• ACL for control traffic
– To redirect control traffic to CPU, not user-configurable
– CoPP
53
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
ACL Scaling and TCAM Partition
• 4096 Access Control Entries per UPC are organized in blocks of 64 ACE entries.
• Each IPv6 ACL (without port range) requires two TCAM entries.
54
switch# sh platform afm info tcam 0 region ifacl
ifacl tcam TCAM configuration for asic id 0:
[ vacl tcam]: range 0 - 1023
[ifacl tcam]: range 1024 - 2175 *
[ qos tcam]: range 2176 - 2623
[rbacl tcam]: range 2624 - 3775
[ span tcam]: range 3776 - 3839
[ sup tcam]: range 3840 - 4095
TCAM [ifacl tcam]: [v:1, size:1152, start:1024
end:2175]
In use tcam entries: 24
1024-1039,2168-2175
Default TCAM Partition
VACL(1024)
IFACL(1152)
QoS(448)
RBACL(1024)
SPAN(64)
Control Traffic(256)
40
96
Entr
ies
TCAM Region
Usage
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
• TCAM Carving allows to change the size of each region
• Create a template:
switch(config)# hardware profile tcam resource template t1
switch(config-tcam-templ)#
• Set region sizes:
switch(config-tcam-templ)# ifacl ?
<320-3584> Enter size of ifacl region (in 64 entry
increments)
TCAM Carving
55
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
TCAM Carving
• Similarly, other region sizes can be changed: switch(config-tcam-templ)# ?
ifacl Configure size of ifacl region
no Change size to default value
qos Configure size of qos region
rbacl Configure size of rbacl region
span Configure size of span region
vacl Configure size of vacl region
• Display the configuration for a template along with the current usage: switch(config)# show hardware profile tcam resource template name
<my_template>
56
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
TCAM Carving
switch(config-tcam-templ)# show hardware profile tcam resource template name t1
-------------------------------------------------------------------
Template name: t1
Current state: Created
Region Size-allocated Current-size Current-usage Available/free
-------------------------------------------------------------------
Vacl 1152 1024 28 996
Ifacl 1024 1152 266 971
Rbacl 1152 1152 3 1149
Qos 448 448 30 418
Span 64 64 6 58
Sup 256 256 74 182
-------------------------------------------------------------------
57
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
TCAM Carving
• Commit a template: switch(config)# hardware profile tcam resource service-template t1
Details of the t1 template you are trying to commit are as follows:
-------------------------------------------------------------------------------
Template name: t1
Current state: Committed (Startup/Running)
Region Features Size-allocated Current-size Current-usage Available/free
-------------------------------------------------------------------------------
Vacl Vacl 1152 2048 11 2037
[…]
------------------------------------------------------------------------------
Template committed.
Please do the following for the template to be applied:
1> Save running config : "copy running-config startup-config"
2> Reboot the switch : "reload”
58
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
59
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Why Analytics? – A Data Center Today:
• 1G -> 10G -> 40G -> 100G
• Virtualized
• More and more applications
• Those applications are more and more complex (Hadoop…)
• More and more flows
• Latency is very important
• Ports speed mismatch
• Buffer discrepancy between the devices
• With spine/leaf, design is less complex, but there’s more speed and more applications
60
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Switched Port Analyzer (SPAN)
61
Monitor Source
Sniffer Device
Monitor Destination
Host-A Host-B
Switch
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Encapsulated Remote SPAN (ERSPAN)
62
Host-A
Host-B
Sniffer Device
Monitor Source Monitor Destination
IP Cloud
GRE Encapsulated
monitored traffic
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 SPAN
63
*Software roadmap feature
SPAN Features Nexus 5600/6000 Nexus 5500
Total SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional
Local SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional
ERSPAN sessions 16 4
Prioritize data over SPAN Yes (through scheduling) Yes (SPAN policing)
Line-rate SPAN throughput Yes No
ERSPAN destination session Yes No
ERSPAN with 1588 PTP timestamp Yes* No
MTU Truncated SPAN and ERSPAN Yes Yes
ACL filter for SPAN and ERSPAN Yes Yes
SPAN-on-Drop Yes No
SPAN-on-Latency Yes No
SPAN with multiple destination ports Yes (each destination port uses one
SPAN session)
No
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
High-Performance SPAN
64
Unified
Crossbar
Fabric
Scheduler Prioritizes Production Data
Data
SPAN
Extra Fabric Bandwidth for SPAN
Separate Buffer Pool for SPAN
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN Buffer Monitoring
Station
Ingress
Data Buffer
SP
AN
-On
-Dro
p
Tail-
Dro
p
SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop
65
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014
• Copies the dropped packet to capture device
• Correlate the packet drop with application
• Works for unicast packets only
• Packet drops can be monitored on ingress only
• There is a dedicated buffer space for SPAN, so SPAN-on-Drop does not affect production traffic
• Supports both local SPAN and ERSPAN
• Configuration uses existing SPAN commands
• One SPAN-on-Drop session is supported
• Can have multiple source ports, and multiple destination ports
• Source cannot be FEX HIF port. But FEX fabric port is supported
• Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously
SPAN-on-Drop
66
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014
The source interface is the ingress port for which we want to monitor drops.
switch(config)# monitor session 1 type span-on-drop
switch(config-span-on-drop)# source interface e3/1
switch(config-span-on-drop)# destination interface e3/2
switch(config)# monitor session 2 type span-on-drop-erspan
switch(config-span-on-drop-erspan)# source interface e3/1
switch(config-span-on-drop-erspan)# destination ip 100.1.1.2
SPAN-on-Drop
67
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Latency Monitoring
Latency Monitoring:
Min: 856ns
Max: 1208ns
Avg: 901ns
TimeStamp
Packet
T0
T1 – T0
T1 – T0
T0
Latency Histogram:
Latency Range: 800ns - 10usec
Stats:
Within the Range: x packets
Out of the Range: y packets
• FIFO Latency
• Per Port-Pair Data
68
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Latency Monitoring - NX-OS 7.0(1)N1(1), Q1/2014
• Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanosecs)
• By default instantaneous Latency Monitoring is enabled between pair of ports
• Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode
• Measures switch latency for each packet, no sampling required
• Fully implemented in HW, no CPU impact, no traffic impact
Latency Monitoring
69
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Instantaneous Mode
• Enabled by default on all pairs of ports. No configuration is required
switch# show hardware profile latency monitor interface e1/23 interface e1/22 -------------------------------------------------------------------------------- Latency Statistics in nano seconds -------------------------------------------------------------------------------- | Ingress Port | Egress Port | Minimum | Maximum | Average | -------------------------------------------------------------------------------- | Ethernet1/22| Ethernet1/23| 856| 1208| 901| --------------------------------------------------------------------------------
70
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Custom Histogram
• Count the number of packets falling in a specific range of latency
• Example for port-pair e1/1 and e1/3:
switch# show hardware profile latency monitor histogram interface e1/3 interface e1/1 -------------------------------------------------------------------------------- | Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/1 | -------------------------------------------------------------------------------- | All Latency Values are in nano seconds | -------------------------------------------------------------------------------- | Range | 800 <= Latency < 10000 | Outside the first range | ------------------------------------------------------------------------------- | counter | 3542818572 | 16290988 |
switch(config)# interface e1/3
switch(config-if)# packet latency low-latency 800 high-latency 10000 mode custom
interface e1/1
71
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN-on-Latency
Latency Monitoring:
Min: 856ns
Max: 1208ns
Avg: 901ns
TimeStamp
Packet
T0
T1 – T0
T3 – T2
T2
Monitoring
Station
If Latency Threshold > 10 usec:
SPAN to 1/64
72
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN-on-Latency
• This feature replicates packets from the interface when latency exceeds a configured threshold
• Latency threshold is per-port
• This again helps to identify the victim of congestion
• Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic
• One SPAN-on-Latency session is supported in hardware
• Uses similar SPAN CLI, with new session type “span-on-latency”
• ERSPAN-on-latency is also supported
73
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN-on-Latency Configuration
monitor session 1 type span-on-latency
source interface Ethernet1/3 tx
destination interface Ethernet1/4
• The SPAN-on-Latency source port is the egress port on which we monitor the latency
• The SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port
• Sample configuration to SPAN packets if latency goes beyond 10us when egressing on e1/3:
interface Ethernet1/3
hardware profile latency monitor threshold 10001
interface Ethernet1/4
switchport mode monitor
Always Tx: packets egressing on 1/3
(any source) with latency >10us will
be replicated to the SPAN dest 1/4
74
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
SPAN-on-Latency
• Source port can be an regular ethernet port, not a port-channel. Can be a port-channel member
• Source port cannot be FEX HIF port. But FEX fabric port is supported
• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX source port
• A SPAN-on-Latency source port cannot be in another SPAN session
• Destination is only a single ethernet port, not port-channel
• Only one destination port can be configured
75
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
76
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Multicast Features Summary
• IGMP snooping for IGMPv1, v2, and v3
• PIM-SM
• PIM-SSM without vPC and PIM-SSM with vPC+
• PIM-BiDir without vPC and PIM-BiDir with vPC+
• PIM Policy
• MSDP
• Anycast RP with PIM (RFC 4610)
• Anycast RP with MSDP
• 8000 IGMP snooping entries
• 8000 mroutes with vPC, 16000 mroutes without vPC supported by software
77
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Efficient Multicast Replication
• Optimized multicast replication
• Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver—minimizing the traffic load on the switch fabric and eliminating the switch fabric congestion
• Line-rate multicast replication
78
Ingress UPC
Unified
Crossbar
Fabric
Egress
UPC
Egress
UPC
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Multicast VOQ • 8000 multicast VOQs to eliminate HOLB and help ensure high throughput
• Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VOQs so that it can be scheduled and served independently
79
Unified
Crossbar
Fabric
Ingress UPC
Egress UPC 1 224.1.1.1
224.1.1.1
224.1.1.2
224.1.1.2
224.1.1.2
Egress UPC 2
Egress UPC 3
Un
ica
st
VO
Q
Mu
ltic
as
t V
OQ
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 Key Forwarding Tables
80
Host table: 256,000 -entry hashing table;
actual capacity is slightly less than 256,000
Host table: Shared between MAC, ARP, and
ND and /32 host route
Host table default carving: 128,000 MAC,
128,000 IP hosts
LPM table: 32,000 entries. Also known as
summary routes
• Mroute table: 64,000 entries
IP Host Region (ARP/ND/Host
route/(*,G))
MAC Region Summary Routes (S,G)
Host Table* LPM Table(32,000) Mroute Table(64,000)*
25
6,0
00
En
trie
s
* Hardware table size. Please check configuration limit for software scaling
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
MET Table
• 64,000 entries in MET table
• Currently supported: 8,000 vPC, 16,000 non-vPC
• Today the MET table is the limiting factor for multicast
scale
Ingress UPC
Egress UPC 1
Un
ica
st
VO
Q
Mu
ltic
as
t V
OQ
Egress UPC 2
1
2
10
Switch Fabric
MET: 123
MET: 123
MET: 123
123 122
16384 16385
65536
1, 4, 5
123 122
16384 16385
65536
2, 10
4
5
81
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Multicast Hashing over Port Channel • N5600/6000 implements flow based hashing for multi-destination traffic, and it supports multicast load
sharing over Port Channel with 5-tuple packet header.
• Traffic is replicated to all egress UPCs where Port Channel member resides
• Egress UPC runs hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example)
82
Egress UPC 1
Egress UPC 2
Port 1
Port 2
Port 3
Port 4
Port 3
Selected
Port 3
Selected
Unified
Crossbar
Fabric
Ingress UPC
Un
ica
st
VO
Q
Mu
ltic
as
t V
OQ
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
83
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 QoS Features
• Eight classes of service;
– 2 reserved for control traffic, 6 for data traffic
• Traffic classification
– DSCP, CoS, and ACL
• Strict Priority Queuing and DWRR
– DCBX 802.1Qaz
• Packet marking
– DSCP, CoS, and ECN
• Ingress and egress policing
– 4096 policers per ASIC
• No drop system class
• Flexible buffer management 84
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Cisco Nexus 5600/6000 QoS Processing Flow
85
Cro
ssb
ar
Fab
ric
MAC Traffic
Classification
Ingress
Cos/DSCP
Marking
MTU
Checking
VOQs for Unicast (8 per egress port)
Egress Queues PAUSE ON/OFF
Signal
Trust CoS/DSCP
L2/L3/L4 info with ACL
Truncate or Drop
Packets if MTU is Violated
If Buffer Usage Crosses Threshold:
• Tail drop for drop class
• Assert pause signal to MAC
for no-drop system class
Strict Priority +
DWRR Scheduling
Multicast Queues
MAC
Ingress UPC
Unicast
Multicast
Ingress
Policing
ECN
Marking Egress
Policing
Egress
Scheduling
Egress UPC
Per-class
Buffer Usage
Monitoring
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Increased Packet Buffer
• 25MB packet buffer is shared by every three 40GE ports or twelve 10GE ports.
• Nexus 5672: 150MB of total buffer
• Nexus 56128: 300MB of total buffer
• Nexus 6004: 800MB of total buffer
86
Ingress UPC Egress UPC
Un
ica
st
VO
Q
Mu
ltic
as
t V
OQ
Unified
Crossbar
Fabric
16MB 9MB
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Ingress Buffer - Flexible Buffer Management
• Shared buffer is good for burst absorption.
• Dedicated buffer is good for predictable performance for each port.
• N5600/6000 buffer management is flexible: dedicated plus shared.
• Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage
87
Port 1
Dedicated
Port 2
Dedicated
Port 3
Dedicated
Shared Packet Buffer
~14MB
SPAN Control
UPC Ingress Buffer (16MB)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Tune Shared vs Dedicated Buffer Allocation at Ingress • “queue-limit” under “network-qos” policy specifies the dedicated buffer for each port and each class.
• Without “queue-limit” each class of service will get 100 KB of dedicated buffer.
• The size of dedicated buffer can be different for different classes of service. The policy applies to all ports
in the chassis.
• Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared
buffer pool.
• The following example sets the dedicated buffer for “class-default” to be 400 KB for all ports.
88
switch(config)# policy-map type network-qos Policy-buffer
switch(config-pmap-nq)# class type network-qos class-default
switch(config-pmap-nq-c)# queue-limit 400000 bytes
switch(config-pmap-nq-c)# system qos
switch(config-sys-qos)# service-policy type network-qos Policy-buffer
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Egress Buffer Design
• 9-MB buffer is reserved for egress traffic
• Unicast traffic can be buffered at egress and ingress.
• Multicast is buffered at egress in case of interface oversubscription.
89
Multicast Buffer Shared ~6MB
UPC Egress Buffer (9MB)
Unicast Buffer Dedicated per port
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Unicast Flow Control and Buffering • Unicast is queued at egress buffer pool first and then ingress buffer after egress queue
is full.
• Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption.
• Ensure fairness among multiple ingress ports
90
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Identify Packet Drops Due to Congestion
91
switch# show interface ethernet 1/7 Ethernet1/1 is down (SFP not inserted)
Dedicated Interface
<snip>
RX <snip>
0 input error 0 short frame 0 overrun 0 underrun 0
ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down
drop
0 input with dribble 235847488 input discard 0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast
packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause
switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 100
RX Queuing
qos-group 0
q-size: 100160, HW MTU: 1500 (1500 configured)
drop-type: drop, xon: 0, xoff: 0
Statistics:
Pkts received over the port : 710025080
Ucast pkts sent to the cross-bar : 474177592
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress :
235847488 Per-priority-pause status : Rx
(Inactive), Tx (Inactive)
Broadcast/Multicast/Unknown unicast flooding drop at egress
Unicast drop at ingress
Ingress drop for each class
Dedicated buffer for this class
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Identify Congested Egress Port with VOQ Counters
92
I
X
I
A
1/7
I
X
I
A
Ingress UPC 3
Egress UPC 3
Ingress UPC 4 I
X
I
A
1/10
1/8
Unified Crossbar
Fabric
Using “show interface” CLI, we noticed drops on ingress ports
How do we know which egress port is affected?
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Step 1: Map Interface to UPC Number
93
N6004# show hardware internal bigsur all-ports
Bigsur Port Info:
Port |asic|inst|inst|
name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer
---------+----+----+----+-----+----+----+---+---+--------+----+-----
sup1 |0 |17 |0 | 0 - |0 |b3 |en |dn |15010000|pass| 0.00
sup0 |0 |17 |0 | 1 - |1 |b3 |en |dn |15020000|pass| 0.00
40gb1/1 |1 |0 |0 | 0 - |0 |b3 |en |dn |1a070000|pass| 0.00
40gb1/2 |1 |0 |0 | 1 - |1 |b3 |en |dn |1a071000|pass| 0.00
40gb1/3 |1 |0 |0 | 2 - |2 |b3 |en |dn |1a072000|pass| 0.00
40gb1/4 |2 |0 |1 | 0 - |3 |b3 |en |dn |1a073000|pass| 0.00
40gb1/5 |2 |0 |1 | 1 - |4 |b3 |dis|dn |1a074000|pass| 0.00
40gb1/6 |2 |0 |1 | 2 - |5 |b3 |dis|dn |1a075000|pass| 0.00
40gb1/7 |3 |0 |2 | 0 p |6 |b3 |en |up |1a076000|pass| 0.00
40gb1/8 |3 |0 |2 | 1 p |7 |b3 |en |up |1a077000|pass| 0.00
40gb1/9 |3 |0 |2 | 2 p |8 |b3 |en |up |1a078000|pass| 0.00
40gb1/10 |4 |0 |3 | 0 p |9 |b3 |en |up |1a079000|pass| 0.00
40gb1/11 |4 |0 |3 | 1 - |10 |b3 |dis|dn |1a07a000|pass| 0.00
40gb1/12 |4 |0 |3 | 2 p |11 |b3 |en |up |1a07b000|pass| 0.00
Done.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Step 2: Identify Congested Egress Port with VOQ Counters
94
N6004# show platform software qd info counters voq asic-num 4 +----------+------------------------------+------------------------+-----------+
| port| TRANSMIT| TAIL DROP| HEAD DROP|
+----------+------------------------------+------------------------+-----------+
Eth1/8
QUEUE-3 2574819434 2248180087 0
+----------+------------------------------+------------------------+-----------+
N6004# show platform software qd info counters voq asic-num 3 +----------+------------------------------+------------------------+-----------+
| port| TRANSMIT| TAIL DROP| HEAD DROP|
+----------+------------------------------+------------------------+-----------+
Eth1/8
QUEUE-3 3888954496 2248133231 0
+----------+------------------------------+------------------------+-----------+
Congested egress port and class of service
Ingress
UPC number
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Architecture - Agenda
95
Overview
Fabric Extender
Performance
Internal Architecture
Transceivers/Cables
Access Control List
Multicast
Quality of Service
Analytics
Applications
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Applications Compact Aggregation
96
CORE
AGG.
ACCESS
Nexus 7000
Nexus 7000
Nexus 6004
Nexus 6004/
Nexus 5600
Nexus 2000 FEX
L3
vPC to Nexus 7000 or 6004
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Applications Large Scale Fabric (Layer 2 or Layer 3)
97
CORE
SPINE
LEAF
Nexus 7000
Nexus 6004
Nexus 6004/
Nexus 5600+2000
L3
FabricPath
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Applications High Performance Computing (HPC)
98
CORE/AGG.
ACCESS
Nexus 6004
Nexus 3548
Nexus 5600/6004
L3/L2
Fabric
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 6004X Chassis
• Nexus 6004X has 8 slots (similar to the 6004EF chassis)
• Nexus 6004X chassis supports VXLAN functionality
• Provides Flexible Ordering Options – mix/match 10/40G LEMs
• Supports new 12 x 40G LEM VXLAN capable (backward compatible with Nexus 6004EF chassis)
• VXLAN capable LEM will work in non-VXLAN mode in 6004EF chassis
• Same software as on Nexus 6004
• Supports all the optics, FEX models as on Nexus 6004
Target Q3 CY14
99
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 5600/6000 Key Takeaways
100
Performance
Rich Feature set
Line rate L2 and L3 with low latency
STP/vPC/FabricPath/DFA, FEX, L3, VXLAN,
Adapter-FEX/VM-FEX
Scalability 256K MAC / 128K ARP / 32K LPM routes,
16K bridge domains with segment ID
Innovations Intelligent SPAN, Buffer monitoring,
Latency monitoring
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Nexus 2000/5600/6000 More Information
101
• http://cisco.com/go/nexus2000
• http://cisco.com/go/nexus5000
• http://cisco.com/go/nexus6000
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Complete Your Online Session Evaluation
• Give us your feedback and you could win fabulous prizes. Winners announced daily.
• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
102
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
103