Post on 24-Nov-2015
Cisco Nexus 7000 Switch ArchitectureBRKARC-3470
Ron Fuller, CCIE#5851 (R&S/Storage)Technical Marketing Engineer
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 3
Session Abstract
This session presents an in-depth study of the architecture of the latest generation of Nexus 7000 and Nexus 7700 data centre switches. Topics include supervisors, fabrics, I/O modules, forwarding engines, and physical design elements, as well as a discussion of key hardware-enabled features that combine to implement high-performance data centre network services.
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 4
Session Goal
To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions
This session will examine the Nexus 7700 system, as well as the latest additions to the Nexus 7000
This session will not examine NX-OS software architecture or other Nexus platform architectures
4
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 5
What Is Nexus 7000?
Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection
Nexus 7000 designed for general-purpose Data Centre deployments, focused on 10G density plus 40G/100G
I/O Modules
Supervisor Engines
Fabrics
Chassis
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 6
What Is Nexus 7700?
Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection
Nexus 7700 designed for SP and MSDC Data Centre deployments, focused on high-density 40G/100G
I/O Modules
Supervisor Engine
FabricsChassis
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 7
Nexus 7000General purpose DC switching w/10/40/100G
Nexus 7700Targeted at Densest 40G/100G deployments
Com
mo
n F
oundatio
n
Same release vehicles, versioning, feature-sets Common configuration model Common operational model
Common fabric ASICs (Fab2) and architecture Same central arbitration model Same VOQ/QoS model
Identical forwarding ASICs (F2E, F3) Consistent hardware feature sets Parallel evolution of hardware capability/scale
Nexus 7000 / Nexus 7700 Common Foundation
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 8
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 9
Nexus 7000 Chassis Family
Front Rear
21RU
N7K-C7010
25RU
Front RearN7K-C7018
Front RearN7K-C7009
14RU
NX-OS 4.1(2) and later
NX-OS 5.2(1) and later
Nexus 7010 Nexus 7018
Nexus 7009
Front N7K-C7004
7RU
NX-OS 6.1(2) and later
Rear
Nexus 7004
Front
Back
Side Side
Side Side Side
Back
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 10
Nexus 7700 Chassis Family
Front Rear
26RU
N77-C7718
Nexus 7718
Front Rear
14RU
N77-C7710
Nexus 7710
Front Rear
9RU
N77-C7706
Nexus 7706
NX-OS 6.2(6) and later
NX-OS 6.2(2) and later
NX-OS 6.2(2) and later
Front
Back
Front
Back
Front
Back
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 11
Key Chassis Components
Nexus 7000
Common components: Supervisor engines
I/O modules
Power supplies (except 7004)
Chassis-specific components: Fabric modules
Fan trays
Nexus 7700
Common components: Supervisor engines
I/O modules
Power supplies
Chassis-specific components: Fabric modules
Fan trays
Common hardware components between Nexus 7000 and Nexus 7700: NONE
No interchangeable hardware components between Nexus 7000 and Nexus 7700
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 12
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 13
Next generation supervisors providing control plane and management functions
Connects to fabric via 1G inband interface
Interfaces with I/O modules via 1G switched EOBC
Second-generation dedicated central arbiter ASIC Controls access to fabric bandwidth via dedicated arbitration path to I/O modules
Supervisor Engine 2 / 2E
Console PortManagement
Ethernet
N7K-SUP2/N7K-SUP2E
USB Host
Ports
ID and Status
LEDs
Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700)
Base performance High performance
One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM
USB Log
Flash
USB Expansion
Flash
N77-SUP2E
ID and Status
LEDs
Console Port Management
Ethernet
USB Expansion
Flash
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 14
Nexus 7000 / 7700 I/O Module Families
M1 1G and 10G
M2 10G / 40G / 100G
F1 10G F2 10GF2E 10G F3 40G
F2E 10GF3 10G / 40G / 100G
F3 closes the
F/M feature gap!
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 15
10G / 40G / 100G M2 I/O modules
Share common hardware architecture
Two integrated forwarding engines (120Mpps) Support for XL forwarding tables (licensed)
Distributed L3 multicast replication
802.1AE LinkSec on all ports
N7K-M224XP-23L
Nexus 7000 M2 I/O ModulesN7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
Supported in NX-OS release 6.1(1) and later
N7K-M206FQ-23L
N7K-M202CF-22L
Module Port Density Optics Bandwidth
M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G
M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G
M2 100G 2 x 100G CFP 200G
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 16
Nexus 7000 M2 I/O Module ArchitectureN7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L
LinkSec +
12 X 10G MAC -or-
3 X 40G MAC -or-
1 X 100G MAC
Forwarding
Engine
VOQs
Fabric 2 ASIC
To Fabric Modules
Replication
Engine
Replication
Engine
Front Panel Ports
LC
CPU
EOBC
VOQs
LinkSec +
12 X 10G MAC -or-
3 X 40G MAC -or-
1 X 100G MAC
Forwarding
Engine
VOQs
Replication
Engine
Replication
Engine
VOQs
To Central Arbiters
Arbitration
Aggregator
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 18
Nexus 7000 / 7700 F2E I/O ModulesN7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E
7000: Supported in NX-OS release 6.1(2) and later
7700: Supported in NX-OS release 6.2(2) and later
N7K-F248XP-25E N7K-F248XT-25E 48-port 1G/10G with SFP/SFP+ transceivers
480G full-duplex fabric connectivity
System-on-chip (SoC) forwarding engine design 12 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)
Interoperability with M1/M2, in Layer 2 mode on Nexus 7000
Proxy routing for inter-VLAN/L3 traffic
LinkSec support* Last 8 ports (SFP+)
All 48 ports (Copper)
Supports Nexus 2000 (FEX) connections
* Roadmap item
N77-F248XP-23E
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 19
Nexus 7000 F2E Module ArchitectureN7K-F248XP-25E / N7K-F248XT-25E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
2 4
LC
CPU
EOBCTo Central Arbiters
Arbitration
Aggregator
4 X 10G
SoC
6 8
4 X 10G
SoC
10 12
4 X 10G
SoC
14 16
4 X 10G
SoC
18 20
4 X 10G
SoC
22 24
4 X 10G
SoC
26 28
4 X 10G
SoC
30 32
4 X 10G
SoC
34 36
4 X 10G
SoC
38 40
4 X 10G
SoC
42 44
4 X 10G
SoC
46 48
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
LinkSec-capable (F2E fibre)
LinkSec-capable (F2E copper)
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 20
Nexus 7700 F2E Module ArchitectureN77-F248XP-23E
4 X 10G
SoC
Front Panel Ports
To Fabric Modules
Fabric 2
2 4
LC
CPU
EOBCTo Central Arbiters
Arbitration
Aggregator
4 X 10G
SoC
6 8
4 X 10G
SoC
10 12
4 X 10G
SoC
14 16
4 X 10G
SoC
18 20
4 X 10G
SoC
22 24
4 X 10G
SoC
26 28
4 X 10G
SoC
30 32
4 X 10G
SoC
34 36
4 X 10G
SoC
38 40
4 X 10G
SoC
42 44
4 X 10G
SoC
46 48
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
LinkSec-capable
Fabric 2
To Fabric Modules
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 22
Nexus 7000 F3 40G Module
12-port 40G QSFP+ module
480G full-duplex fabric connectivity
SoC forwarding engine design 6 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
Fabric Services Accelerator (FSA) CPU
Breakout cable support
Requires Supervisor Engine 2 / 2E
N7K-F312FQ-25
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 23
Nexus 7000 12-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 1
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
Fabric ASIC
LC Inband
2 3 4 5 6 7 8 9 10 11 12
x 6
to FSA
CPUto ARB
x 6
1G switch
x 6
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 24
FSA CPU
Fabric Services Accelerator (FSA) High-performance module CPU
with on-board acceleration engines
6Gbps inband connectivity from SOCs to FSA
Multi-Mpps packet processing
2GB dedicated DRAM
Performance/scale boost for distributed fabric services, including BFD and sampled NetFlow (roadmap)
Other potential applications include distributed ARP/ping processing, data plane packet analysis (wireshark), network probing, etc.
6 x 1Gbps
Module Inband
I/O
2GBDRAM
Dual-CoreLC CPU
AccelerationEngines
2GBDRAM
EOBC
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 25
Nexus 7700 F3 48-Port 1G/10G Module
48-port 1G/10G with SFP/SFP+ transceivers
480G full-duplex fabric connectivity
SoC-based forwarding engine design 6 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
Fabric Services Accelerator (FSA) CPU
LinkSec support (last 8 ports)*
Supports Nexus 2000 (FEX) connections
N77-F348XP-23
* Roadmap item
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 26
8 X 10G
SoC 1
Nexus 7700 F3 48-Port 1G/10G Module Architecture
To Fabric Modules To Central Arbiters
Arbitration
Aggregator
8 X 10G
SoC 2
8 X 10G
SoC 3
8 X 10G
SoC 4
8 X 10G
SoC 5
8 X 10G
SoC 6
Fabric ASIC Fabric ASIC x 6
1
Front Panel Ports (SFP/SFP+)
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
LinkSec-capable
to FSA
CPUto ARB
FSA
CPU
EOBC
LC Inband
x 6
1G switch
x 6
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 27
Nexus 7700 F3 40G and 100G Modules
24-port 40G QSFP+ module / 12-port 100G CPAK module
960G/1.2T full-duplex fabric connectivity
SoC forwarding engine design 12 independent SoC ASICs
Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features
Fabric Services Accelerator (FSA) CPU
40G breakout cable support*
N77-F324FQ-25
N77-F312CK-26
* Roadmap item
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 28
2 X 40G
SoC 1
Nexus 7700 F3 24-Port 40G Module Architecture
1
Front Panel Ports (QSFP+)
To Fabric Modules
FSA
CPU
EOBC To Central Arbiters
Arbitration
Aggregator
2 X 40G
SoC 2
2 X 40G
SoC 3
2 X 40G
SoC 4
2 X 40G
SoC 5
2 X 40G
SoC 6
2 X 40G
SoC 7
2 X 40G
SoC 8
2 X 40G
SoC 9
2 X 40G
SoC 10
2 X 40G
SoC 11
2 X 40G
SoC 12
Fabric ASIC Fabric ASIC
LC Inband
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
1G switch
x 12
to FSA
CPUto ARB
x 12
x 6
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 29
Nexus 7700 F3 12-Port 100G Module Architecture
Front Panel Ports (CPAK)
To Fabric Modules To Central Arbiters
Arbitration
Aggregator
1 X 100G
SoC 2
2
1 X 100G
SoC 3
3
1 X 100G
SoC 4
4
1 X 100G
SoC 5
5
1 X 100G
SoC 6
6
1 X 100G
SoC 7
1 X 100G
SoC 8
1 X 100G
SoC 9
1 X 100G
SoC 10
1 X 100G
SoC 11
Fabric ASIC Fabric ASIC
7 8 9 10 11
1 X 100G
SoC 12
12
1 X 100G
SoC 1
1
FSA
CPU
EOBC
LC Inband
1G switch
x 12
to FSA
CPUto ARB
x 12
x 6
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 30
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 31
M-Series Forwarding Engine Hardware Two hardware forwarding engines
integrated on every M2 I/O module
120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning
120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4
60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast
Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir)
MPLS/VPLS/EoMPLS
OTV
RACL/VACL/PACL
QoS remarking and policing policies
Policy-based routing (PBR)
Unicast RPF check and IP source guard
IGMP snooping
Ingress and egress NetFlow (full and sampled)Hardware Table M-Series Modules
without Scale License
M-Series Modules with
Scale License
MAC Address Table 128K 128K
FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6
Classification TCAM (ACL/QoS) 64K 128K
NetFlow Table 1M 1M
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 32
From I/O Module
Replication Engines
To I/O Module
Replication Engines
M-Series Forwarding Engine Architecture
L2 Engine
Ingress Parser
MAC
Table L2 Lookup (pre-L3)
L2 Lookup (post-L3)
Final Results
L3 Engine
Classification
(ACL/QoS)
NetFlow
Layer 3 FIB
Policing
FIB TCAM/
ADJ
CL TCAM
FE Daughter Card
Ingress lookup pipeline
Egress lookup
pipeline
Egress NetFlow collection
Ingress MAC table lookups Port-channel hash result Ingress IGMP snooping
lookups
FIB TCAM and adjacency table lookups for Layer 3 forwarding
ECMP hashing Multicast RPF check
Ingress policing
Egress MAC lookupsEgress IGMP snooping
lookups
PKT
HDR
Egress ACL/QoS classification
Ingress NetFlow collection
Ingress ACL/QoS classification
Egress policing
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 33
F2E Forwarding Engine Hardware
Each SoC forwarding engine services 4 front-panel 10G ports (12 SoCs per module)
60Mpps per SoC Layer 2 bridging with hardware MAC learning
60Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast
Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)
RACL/VACL/PACL
QoS remarking and policing policies
Policy-based routing (PBR)
Unicast RPF check and IP source guard
IGMP snooping
FabricPath forwarding
FCoE (with Sup2 / Sup2E) Roadmap on Nexus 7700
Ingress sampled NetFlow
Hardware Table Per F2E SoC Per F2E Module
MAC Address Table 16K 192K*
FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6
Classification TCAM (ACL/QoS) 16K 192K*
* Assumes specific configuration to scale SoC resources
* Roadmap item
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 34
F3 Forwarding Engine Hardware
Each SoC forwarding engine services: 8 front-panel 10G ports
2 front-panel 40G ports
1 front-panel 100G port
148Mpps per SoC Layer 2 bridging with hardware MAC learning
148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast
Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)
RACL/VACL/PACL
QoS remarking and policing policies
Policy-based routing (PBR)
Unicast RPF check and IP source guard
IGMP snooping
FabricPath forwarding
Overlay Transport Virtualisation (OTV)
MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE*
Ingress/egress* sampled NetFlow
Hardware Table Per F3 SoC Per F3 Module
MAC Address Table 64K 384K/768K**
FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6
Classification TCAM (ACL/QoS) 16K 96K/192K**
** Assumes specific configuration to scale SoC resources
* Roadmap items
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 35
F3 Forwarding Engine
Decision Engine
Layer 3 Lookups
QoS / ACL
Ingress Parser
MAC
Table
FIB/ADJ
CL
L2 Lookup (post-L3)
Front-panel
To/From Central
Arbiter To Fabric From Fabric
Ingress
Buffer (VOQ)
Virtual output
queues
L2 Lookup (pre-L3)
Egress Parser
F3 SoC
Ingress and egress
forwarding decisions
(L2/L3 lookups,
ACL/QoS, features etc.)
8 x 1/10G OR
2 x 40G OR
1 x 100G per ASIC
Forwarding
tables
1G / 10G / 40G / 100G
1G / 10G / 40G / 100G
capable interface MAC
Egress
BufferEgress fabric
receive buffer
HDR
PKT HDR
PKT
PKT HDR
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 36
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 37
Crossbar Switch Fabric Modules
Provide interconnection of I/O modules
Each installed fabric increases available per-payload slot bandwidth
Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC
Different I/O modules leverage different amount of available fabric bandwidth
Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ
N7K-C7018-FAB-2
N7K-C7010-FAB-2
N7K-C7009-FAB-2
Fabric Module Supported ChassisPer-fabric module
bandwidth
Max fabric
modules
Total bandwidth per
slot
Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot
Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot
N77-C7718-FAB-2
N77-C7710-FAB-2
N77-C7706-FAB-2
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 38
110G
(2 x 55G)
Ingress Module Egress Module
Multistage Crossbar
Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric
Stages 1 and 3 on I/O modules
Stage 2 on fabric modules
1st stage Egress
Module
2nd stage
Ingress
Module
3rd stageFabric ASIC Fabric ASIC Fabric ASIC Fabric ASICFabric ASIC Fabric ASIC
Fabric Modules
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
1
Fabric
ASIC2 3 4 5
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC6
Fabric
ASIC
1.32T
1st stage
3rd stage
550G
110G
(2 x 55G)
1 FabricASIC
2 3 4 5FabricASIC
Fabric
ASIC
Fabric
ASIC
Fabric
ASIC
Fabric Modules
Nexus 7000 Nexus 7700
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 39
110Gbps220Gbps330Gbps440Gbps550Gbps
Local Fabric 2
(480G)
Local Fabric 2
(240G)
I/O Module Capacity Nexus 7000
One fabric:
Any port can pass traffic to any other port in VDC
Three fabrics:
240G M2 module has maximum bandwidth
Five fabrics:
480G F2E/F3 module has maximum bandwidth
Fabric 2 Modules
1Fabric 2
ASIC
2Fabric 2
ASIC
3Fabric 2
ASIC
4Fabric 2
ASIC
5Fabric 2
ASIC
per slot bandwidth
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 40
What About Nexus 7004?
Nexus 7004 has no fabric modules
I/O modules have local fabric with 10 available fabric channels I/O modules connect back-to-back via 8 fabric channels
Two fabric channels borrowed to connect supervisor engines
Sup Slot 2Sup Slot 1
M2/F2E/F3
Module 4
M2/F2E/F3
Module 3
Fabric
ASIC
Fabric 2
ASIC
Fabric 2
ASIC
Fabric
ASIC
2 * 55G
fabric channels
8 * 55G local fabric channels
interconnect I/O modules (440G)
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 41
220Gbps440Gbps660Gbps880Gbps1100Gbps1320GbpsLocal Fab2
#1 (480G)
Local Fab2
#1 (960G)
Local Fab2
#1 (1.2T)
Fab2
#2
Fab2
#2
Fab2
#2
I/O Module Capacity Nexus 7700
One fabric:
Any port can pass traffic to any other port in VDC
Three fabrics:
480G F2E/F3 10G module has maximum bandwidth
Five fabrics:
960G F3 40G module has maximum bandwidth
Six fabrics:
1.2T F3 100G module has maximum bandwidth
per slot bandwidth
Fabric 2 Modules
1Fabric 2
ASICs
2Fabric 2
ASICs
3Fabric 2
ASICs
4Fabric 2
ASICs
5Fabric 2
ASICs
6Fabric 2
ASICs
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 42
Fabric, VOQ, and Arbitration
Crossbar fabric Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules
Virtual Output Queues (VOQs) Provide buffering and queuing for ingress-buffered switch architecture
Central arbitration Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports
Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 43
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 44
Buffering, Queuing, and Scheduling
Buffering storing packets in memory Needed to absorb bursts, manage congestion
Queuing buffering packets according to traffic class Provides dedicated buffer for packets of different priority
Scheduling controlling the order of transmission of buffered packets Ensures preferential treatment for packets of higher priority and fair treatment for
packets of equal priority
Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define buffering, queuing, and scheduling behaviour
Default queuing and network-QoS policies always in effect in absence of any user configuration
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 45
I/O Module Buffering Models
Buffering model varies by I/O module family M-series modules: hybrid model combining ingress VOQ-buffered
architecture with egress port-buffered architecture
F-series modules: pure ingress VOQ-buffered architecture
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 46Egress ModuleIngress Module
VOQ 0
Port ASIC 0
M2 Hybrid Ingress/Egress Buffered
DW
RR
VOQ 1RE 1
RE 0
D
WR
R
Port 1
Port 12F
AB
RIC
VOQ 0
VOQ 1 RE 1
RE 0
Port ASIC 0
Port 1
Port 12
Ingress port buffer Manages congestion of ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
INGRESS QUEUING POLICIES
10G module used as example
Diagram represents halfof each I/O module
8 ingress
queues
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 47Egress ModuleIngress Module
VOQ 0
1
2
3
4
5
6
SP234
VOQ
Buffer
So
urc
e
Priority
Port ASIC 0
M2 Hybrid Ingress/Egress Buffered
DW
RR
VOQ 1RE 1
RE 0
D
WR
R
Port 1
Port 12F
AB
RIC
VOQ 0
VQI 1DW
RR
SP
VQI 6DW
RR
SP
VOQ 1
Sources 7-12 VQIs 7-12
RE 1
RE 0
Port ASIC 0
Port 1
Port 12
Ingress port buffer Manages congestion of ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
Ingress VOQ buffer Manages congestion toward egress
destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
FABRIC-QOS POLICY
10G module used as example
Diagram represents halfof each I/O module
Shared buffer
carved by source
and priority4 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 48Egress ModuleIngress Module
VOQ 0
1
2
3
4
5
6
SP234
VOQ
Buffer
So
urc
e
Priority
Port ASIC 0
DW
RR
VOQ 1RE 1
RE 0
D
WR
R
Port 1
Port 12F
AB
RIC
VOQ 0
VQI 1DW
RR
SP
VQI 6DW
RR
SP
VOQ 1
Sources 7-12 VQIs 7-12
RE 1
RE 0
Port ASIC 0
Port 1
Port 12
DW
RR
SP
DW
RR
SP
Ingress port buffer Manages congestion of ingress forwarding/replication engines, and
congestion toward egress destinations (VQIs)
Buffering / queuing / scheduling
Ingress VOQ buffer Manages congestion toward egress
destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
Egress port buffer Manages congestion at egress
physical interface
Buffering / queuing / scheduling
EGRESS QUEUING POLICIES
10G module used as example
Diagram represents halfof each I/O module
8 egress
queues
M2 Hybrid Ingress/Egress Buffered
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 49
Egress SOCIngress SOC
Ingress VOQ
F2E Ingress Buffered (Nexus 7000)
FA
BR
IC
Egress VOQ
VQI 1DW
RR
PQ
VQI 4DW
RR
PQ
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
1
2
3
4
hi
VOQ
Buffer
lo
hi
lo
hi
lo
hi
lo10G Port 1
10G Port 2
10G Port 3
10G Port 4
10G Port 1
10G Port 2
10G Port 3
10G Port 4
Diagram represents one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
2 or 4 ingress
queues per port 4 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 50
Egress SOCIngress SOC
Ingress VOQ
F3 10G Ingress Buffered (Nexus 7700)
FA
BR
IC
Egress VOQ
VQI 1
VQI 8
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
2
4
6
8
VOQ
Buffer
hi
lo1
3
5
7
10G Port 1
10G Port 3
10G Port 5
10G Port 2
Diagram represents one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
10G Port 4
10G Port 6
10G Port 7
10G Port 8
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
hi
lo
10G Port 1
10G Port 2
10G Port 3
10G Port 4
10G Port 5
10G Port 6
10G Port 7
10G Port 8
DW
RR
PQ
DW
RR
PQ
2 or 4 ingress
queues per port
8 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 51
Egress SOCIngress SOC
Ingress VOQ
F3 40G Ingress Buffered (Nexus 7000)
FA
BR
IC
Egress VOQ
VQI 1DW
RR
PQ
VQI 2DW
RR
PQ
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
1
2
VOQ
Buffer
lo
hi
lo
hi
40G Port 1
40G Port 2
40G Port 1
Diagram represents one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
40G Port 2
2 or 4 ingress
queues per port
4 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 52
Egress SOCIngress SOC
Ingress VOQ
F3 40G Ingress Buffered (Nexus 7700)
FA
BR
IC
Egress VOQ
VQI 1
VQI 2
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
40G Port 1
Diagram represents one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
40G Port 2
DW
RR
PQ
DW
RR
PQ
1
2
VOQ
Buffer
lo
hi
lo
hi
40G Port 1
40G Port 2
2 or 4 ingress
queues per port
8 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 53
Egress SOCIngress SOC
Ingress VOQ
F3 100G Ingress Buffered (Nexus 7700)
FA
BR
IC
Egress VOQ
VQI 1
Ingress VOQ buffer Manages congestion toward egress destinations (VQIs)
Buffering / queuing
Egress VOQ buffer Receives frames from fabric
Scheduling
1VOQ
Buffer
hi
lo
100G Port 1
Diagram represents one SoC on each I/O module
INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES
DW
RR
PQ
100G Port 1
2 or 4 ingress
queues per port
8 priority
levels
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 54
FAQ: What Is a VQI?
VQI = Virtual Queuing Index
A Destination Across the Fabric
For M2 / F2E / F3 10G modules, VQI == 10G interface
For M2 40/100G ports, uses multiple 10G VQIs
For F3 40/100G ports, uses single 40/100G VQI
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 55
40G Port
Ingress Modules
10G 10G 40G 40G 100G
SpinesSpines
SpinesSpinesFabrics
M2 Module 40G and 100G Flow Limits
Each Virtual Queuing Index (VQI) sustains 10G traffic flow
All packets in given 5-tuple flow hash to single VQI
Single-flow limit is 10G
Packets split into 66-bit code words
Four code words transmitted in parallel, one on each physical Tx fibre
No per-flow limit imposed splitting occurs at physical layer
Egress Interfaces
Destination
VQIs
1 VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs
Internal to Nexus 7000 System
n 4 3 2 1
64 bits
1 packet
On the Wire (40G)
Tx 1
Tx 2
Tx 3
Tx 4
66 bits
15
2
3
4
6
64
/66
B E
nc
od
ing
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 56
Ingress Modules
10G 10G 40G 40G 100G
SpinesSpines
SpinesSpinesFabrics
F3 Module 40G and 100G Flow Limits
Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type
No single-flow limit full 40G/100G flow support
Egress Interfaces
Destination
VQIs
1 VQI 1 VQI 1 VQI 1 VQI 1 VQI
Internal to Nexus 7000 / 7700 System
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 57
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 58
Hardware Layer 2 Forwarding Process
Layer 2 forwarding traffic steering based on destination MAC address
MAC table lookup drives Layer 2 forwarding
Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs
Source MAC lookup drives new learns and refreshes aging timers
Destination MAC lookup dictates outgoing switchport
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 59
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L2 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
ACL/QoS/
NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result destination +
hash result
Credit grant for
fabric access
Egress
port QoS LinkSec
encryption
Static or hash-
based RE uplink
selection
Hash-based uplink
and VQI selection
Round-robin
transmit to VQI
Static
downlink
selection
L2 SMAC/ DMAC
lookups
Port-channel hash
result
HDR = Packet Headers DATA = Packet Data = Internal SignallingCTRL
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 60
SoC
VOQ
SoC
DE
F2E / F3 L2 Packet Flow
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
VOQ arbitration
Credit grant for
fabric access
Receive from fabric
Return
credit
to pool
Transmit
packet on
wire
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
Transmit
to fabric
VOQ
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Ingress L2 SMAC/ DMAC
lookups, ACL/QoS lookups,
NetFlow sampling Return result
destination
Submit packet headers for lookup
Egress port QoS
(Scheduling)
Return buffer credit
HDR = Packet Headers DATA = Packet Data = Internal SignallingCTRL
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 61
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 62
Layer 3 Forwarding
Nexus 7000 decouples control plane and data plane
Forwarding tables built on control plane using routing protocols or static configuration
OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing
Tables downloaded to forwarding engine hardware for data plane forwarding FIB TCAM contains IP prefixes
Adjacency table contains next-hop information
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 63
Hardware Layer 3 Forwarding Process
FIB TCAM lookup based on longest-match destination prefix comparison
FIB hit returns adjacency, adjacency contains rewrite information (next-hop)
Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 64
10.1.1.2
10.1.1.3
10.10.0.10
10.10.0.100
10.10.0.33
10.1.1.4
10.1.2.xx
10.1.3.xx
10.1.1.xx
10.100.1.xx
10.10.0.xx
10.100.1.xx
10.10.100.xx
IP FIB TCAM Lookup
FIB TCAM
Generate
Lookup Key
10.1.1.10
Generate TCAM lookup key
(destination IP address)
Forwarding Engine
FIB DRAM
Load-SharingHash
Adjacency Table
Next-hop 4 (IF, MAC)
Next-hop 6 (IF, MAC)
Next-hop 7 (IF, MAC)
Next-hop 5 (IF, MAC)
Next-hop 3 (IF, MAC)
Next-hop 1 (IF, MAC)
Next-hop 2 (IF, MAC)
10.1.1.xx
Ingress
unicast IP
packet header
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Index, # next-hops
Hit in FIB
returns result
in FIB DRAM
Adjacency
index identifies
ADJ block to
use
Modulo function
selects exact
next hop entry
to use
Offset
Compare
lookup key
Return lookup
result
# next-
hops
Flow
Data
ResultHIT!
Adj Index
mod
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 65
Module 1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e1/1
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
Module 2
Fabric 2 ASIC
10G/40G/100G MAC / LinkSec
VOQs
Replication
Engine
Replication
Engine
VOQs
e2/2
Layer 2
Engine
Layer 3
Engine
Forwarding
Engine
M2 L3 Packet Flow
Receive
packet from
wire
LinkSec decryption
Ingress port QoS
Submit packet
headers for
lookup
L3 FIB/ADJ lookup
Ingress and egress
ACL/QoS/NetFlow
lookups
VOQ arbitration
and queuing
Round-robin
transmit to fabric
Receive from
fabric
Return buffer
credit
Return
credit
to pool
Transmit
packet on
wire
Return result destination +
hash result
Credit grant for
fabric access
Egress
port QoS LinkSec
encryption
Static or Hash-based
uplink selection
Hash-based uplink
(and VQI) selection
Round-robin
transmit to VOQ
Static RE
downlink
selection
L2 ingress and egress
SMAC/ DMAC lookups
Port-channel hash result
HDR = Packet Headers DATA = Packet Data = Internal SignallingCTRL
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 66
SoC
VOQ
SoC
DE
Module 2
Fabric ASIC
e2/2
Module 1
Fabric ASIC
e1/1
Fabric Module 1
Fabric ASIC
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
Supervisor Engine
Central Arbiter
Fabric Module 4
Fabric ASIC
Fabric Module 5
Fabric ASIC
VOQ
F2E / F3 L3 Packet FlowHDR = Packet Headers DATA = Packet Data = Internal SignallingCTRL
VOQ arbitration
Credit grant for
fabric access
Return
credit
to pool
Transmit
packet on
wire
Transmit
to fabric
Receive
packet
from wire
Ingress
port QoS
(VOQ)
Return result destination
Submit packet headers for lookup
L2 ingress and egress SMAC/
DMAC lookups
L3 FIB/ADJ lookup
Ingress and egress ACL/QoS
lookups, NetFlow sampling
Receive from fabric
Egress port QoS
(Scheduling)
Return buffer credit
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 67
Layer 3 Forwarding Module Interoperability Models
Two interoperability models for L3 forwarding:
Proxy Forwarding
Ingress Forwarding with Lowest Common Denominator
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 68
From F1/F2E perspective, Router MAC reachable through giant port-channel
All packets destined to Router MAC forwarded through fabric toward one member port in that channel
Proxy Forwarding Model Conceptual
All F1/F2E modules
All M1/M2 modules
Up to 128 links10.1.10.100vlan 10
10.1.20.100vlan 20
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 69
Proxy Forwarding Model Actual
10.1.10.100vlan 10
e1/1Fabric
F1/F2E
SoC
FE
e2/1Fabric
F1/F2E
SoC
FE
10.1.20.100vlan 20
Replication
Engine
e3/1
e3/2
M1/M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e3/3e3/4
e3/5
e3/6
e3/7e3/8
Replication
Engine
e4/1
e4/2
M1/M2
Replication
Engine
Replication
Engine
Replication
Engine
VOQs
VOQs
FE
FE
Fabric
e4/3
e4/4
e4/5
e4/6
e4/7
e4/8
Fabric
FabricModules
Fabric
VLAN DMAC Dest Port
10 router_mac internal_channel (e3/1-8,e4/1-8)
EtherChannel Hash Function
hash_input (from packet) select_member_port
Ingress MAC:
VLAN DMAC Dest Port
10 router_mac L3_lookup
Routing:
DIP Next Hop
10.1.20.100 server_2_mac (v20)
Egress MAC:
VLAN DMAC Dest Port
20 server_2_mac e2/1
1
2
3
4
6
5 7
8
9
10
Programming of all M1/M2 forwarding engines
Programming of all F1/F2E forwarding engines
interface vlan 10
ip address 10.1.10.1/24
!
interface vlan 20
ip address 10.1.20.1/24
Can be up to 128 M1/M2 VQIs
Mod 1
Mod 2
Mod 4
Mod 3
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 71
Ingress Forwarding with Lowest Common Denominator Model
F3 module interoperability always Ingress Forwarding NO proxy forwarding with F3
Essentially equivalent to current M1 + M2 interoperability model
The ingress module makes all the forwarding decisions
Supported feature set based on Lowest Common Denominator Feature available if all modules support the feature
VDC Type Layer 2 Layer 3 vPCFabric
PathVXLAN FEX MPLS OTV LISP FCoE Table Sizes
F3 F3 size
M2 + F3 F3 size
F2/F2E + F3 F2E size
Not all features
supported by
software today
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 73
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 74
What Is Classification?
Matching packets Layer 2, Layer 3, and/or Layer 4 information
Used to decide whether to apply a particular policy to a packet Enforce security, QoS, or other policies
Some examples: Match TCP/UDP source/destination port numbers to enforce security policy
Match destination IP addresses to apply policy-based routing (PBR)
Match 5-tuple to apply marking policy
Match protocol-type to apply Control Plane Policing (CoPP)
etc.
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 75
CL TCAM Lookup ACL ip access-list examplepermit ip any host 10.1.2.100
deny ip any host 10.1.68.44
deny ip any host 10.33.2.25
permit tcp any any eq 22
deny tcp any any eq 23
deny udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
xxxxxxx | 10.1.2.100 | xx | xxx | xxx
xxxxxxx | 10.1.68.44 | xx | xxx | xxx
xxxxxxx | 10.33.2.25 | xx | xxx | xxx
xxxxxxx | xxxxxxx | tcp | xxx | 22
xxxxxxx | xxxxxxx | tcp | xxx | 23
xxxxxxx | xxxxxxx | tcp | xxx | 80
xxxxxxx | xxxxxxx | udp | xxx | 161
xxxxxxx | xxxxxxx | udp | xxx | 514
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
CL TCAM
Generate
Lookup Key
Generate TCAM
lookup key
CL SRAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx | 10.2.2.2 | xx | xxx | xxx
xxxxxxx | xxxxxxx | tcp | xxx | 80
SIP | DIP | Pr | SP | DP
Compare lookup
key to CL TCAM
entries
Comparisons(X = Mask)
Hit in CL TCAM
returns result in
CL SRAM
Security ACL
Forwarding Engine
Result
Return
lookup
result
Result affects
final packet
handling
Permit
Permit
Permit
Permit
Deny
Deny
Deny
Deny
HIT!
Results
SIP | DIP | Pr | SP | DP
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 76
Packet header:
SIP: 10.1.1.1
DIP: 10.2.2.2
Protocol: TCP
SPORT: 33992
DPORT: 80
Result affects
final packet
handling
Generate
Lookup Key
Forwarding Engine
xxxxxxx | 10.3.3.xx | xx | xxx | xxx
xxxxxxx | 10.4.12.xx | xx | xxx | xxx
10.1.1.xx | xxxxxxx | udp | xxx | xxx
10.1.1.xx | xxxxxxx | tcp | xxx | xxx
xxxxxxx | 10.5.5.xx| tcp | xxx | 23
CL TCAM Lookup QoS ip access-list policepermit ip any 10.3.3.0/24
permit ip any 10.4.12.0/24
ip access-list remark-dscp-32
permit udp 10.1.1.0/24 any
ip access-list remark-dscp-40
permit tcp 10.1.1.0/24 any
ip access-list remark-prec-3
permit tcp any 10.5.5.0/24 eq 23
CL TCAM
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80
xxxxxxx | 10.2.2.xx | xx | xxx | xxx
10.1.1.xx | xxxxxxx | tcp | xxx| xxx
HIT!
CL SRAM
QoS Classification ACLs
Generate
TCAM lookup
key
SIP | DIP | Pr | SP | DP
Compare
lookup
key
Hit in CL TCAM
returns result in
CL SRAM
Result
Return
lookup
result
Policer ID 1
Policer ID 1
Remark DSCP 32
Remark DSCP 40
Remark IP Prec 3
SIP | DIP | Pr | SP | DP
Comparisons(X = Mask)
Results
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 77
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 78
NetFlow
NetFlow collects flow data for packets traversing the switch
Each module maintains independent NetFlow table
M2 F2E / F3
Per-interface NetFlow Yes Yes
NetFlow direction Ingress/Egress Ingress only
Full NetFlow Yes No
Sampled NetFlow Yes Yes
FSA Assist for Sampled NetFlow No F3 only (future)
Bridged NetFlow Yes Yes
Hardware Cache Yes No
Software Cache No Yes
Hardware Cache Size512K entries per
forwarding engineN/A
NDE (v5/v9) Yes Yes
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 79
Full vs. Sampled NetFlow
NetFlow collects full or sampled flow data
Full NetFlow: Accounts for every packet of every flow on interface Available on M-Series modules only
Flow data collection up to capacity of hardware NetFlow table
Sampled NetFlow: Accounts for M in N packets on interface Available on both M2 (ingress/egress) and F2E/F3 (ingress only)
M2: Flow data collection up to capacity of hardware NetFlow table
F2E/F3: Flow data collection for up to ~1000pps per module
F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 81
NetFlow on M2 Modules
Fabric
ASIC
VOQs
MgmtEnet
Supervisor
Engine
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Forwarding
Engine
LC
CPU
NetFlow
Table
M2 Module
Hardware
Flow Creation
Hardware
Flow Creation
Hardware
Flow Creation
Aged Flow Info
Aged Flow Info
Aged Flow Info
Generate NetFlow v5
or v9 export packets
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 82
Sampled NetFlow on F2E/F3 Modules
F3 Module
FSA
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
F3 Module
Fabric
ASIC
VOQs
MgmtEnet
Supervisor
Engine
FSA
CPU
SoC
Decision
Engine
Main
CPU
To NetFlow Collector
To NetFlow Collector
Switched
EOBC
via Supervisor
Inband
via mgmt0
DRAM
NetFlow
Cache
Populate cache based
on received samples
Age flows and
generate NetFlow v5
or v9 export packets
F2E Module
LC
CPU
SoC
Decision
Engine
DRAM
NetFlow
Cache
Data Flow
Data Flow
Data Flow
via Module
Inband
via Module
Inband
via Module
Inband
Sampled
Packets
Sampled
Packets
Sampled
Packets
Aged
Flows
Aged
Flows
Aged
Flows
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 83
Agenda
Chassis Architecture
Supervisor Engine and I/O Module Architecture
Forwarding Engine Architecture
Fabric Architecture
I/O Module Queuing
Layer 2 Forwarding
Layer 3 Forwarding
Classification
NetFlow
Conclusion
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 84
Nexus 7000 / Nexus 7700 Architecture Summary
I/O Modules
Supervisor Engines
Fabrics
Chassis
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 85
Conclusion
You should now have a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, I/O module design, packet flows, and key forwarding engine functions
Any questions?
85
Q & A
2014 Cisco and/or its affiliates. All rights reserved.BRKARC-3470 Cisco Public 88
Complete Your Online Session Evaluation
Give us your feedback and receive a Cisco Live 2014 Polo Shirt!
Complete your Overall Event Survey and 5 Session Evaluations.
Directly from your mobile device on the Cisco Live Mobile App
By visiting the Cisco Live Mobile Site www.ciscoliveaustralia.com/mobile
Visit any Cisco Live Internet Station located throughout the venue
Polo Shirts can be collected in the World of Solutions on Friday 21 March 12:00pm - 2:00pm
Learn online with Cisco Live!
Visit us online after the conference for full access
to session videos and presentations.
www.CiscoLiveAPAC.com