1. DataCenter Infrastructure - VXLAN and Nexus 7K Architecture David Palma Solutions Architect SLED North East davpalma@cisco.com

2. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Overlays Introduction to VXLAN VXLAN Design Deployment Steps Key Takeaways References 3. Overlays 4. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Why Overlays? Flexible Overlay Virtual Network Mobility Track end-point attach at edges Scale Reduce core state Distribute and partition state to network edge Multi-tenancy Share Network resources Flexibility/Programmability Reduced number of touch points Robust Underlay/Fabric High Capacity Resilient Fabric Intelligent Packet Handling Programmable & Manageable 5. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Types of Overlay Service Emulate a LAN segment Transport Ethernet Frames (IP and non-IP) Single subnet mobility (L2 domain) Exposure to open L2 flooding Useful in emulating physical topologies Abstract IP based connectivity Transport IP Packets Full mobility regardless of subnets Contain network related failures (floods) Useful in abstracting connectivity and policy Layer 2 Overlays Layer 3 Overlays 6. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Types of Overlay Edge Devices Virtual end-points only Single admin domain VXLAN, NVGRE, STT Tunnel End-points Physical and Virtual Resiliency + Scale x-organizations/federation Open Standards Network Overlays Integrated Overlays A p pO S A p pO S Virtual Physical Fabric DB V M O S V M O S Virtual Virtual V M O S V M O S Host Overlays Physical Physical Router/switch end-points Protocols for resiliency/loops Traditional VPNs OTV, VXLAN, VPLS, LISP 7. Introduction to VXLAN 8. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Overview Challenges VXLAN addresses: VLAN Scalability (4K) VXLAN extends the L2 Segment ID field to 24- bits, potentially allowing up to 16 million unique L2 Segments over the same network VM mobility restricted within a VLAN VXLAN encapsulates L2 frame in IP- UDP header allowing L2 adjacency across router boundaries VXLAN Technology Overview: MAC-in-UDP encapsulation Leverages multicast in the transport network to simulate flooding behavior for broadcast, unknown unicast and multicast in the layer 2 segment Leverage ECMP to achieve optimal path usage over the transport network 9. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Overview VLAN 10 VLAN 20 vSwitch VTEP vSwitch VTEP Switch VTEP VNI 1000 VNI 2000 VXLAN can be implemented on both Hypervisor-based Virtual Switches to allow for scalable VM deployments, as well as on Physical switches, which provides the ability to bridge VXLAN segments back into VLAN segments. In these cases, the Physical Switch instantiates a VTEP, and function as a VXLAN Gateway 10. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Virtual eXtensible LAN (VXLAN) Virtual eXtensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3 network. A 24-bit VXLAN Segment ID or VXLAN Network Identifier (VNI) is included in the encapsulation to provide up to 16M VXLAN segments for traffic isolation / segmentation, in contrast to 4K segments achievable with VLANs. Each of these segments represents a unique Layer 2 broadcast domain, and can be administered in such a way that it can uniquely identify a given tenants address space or subnet. Ethernet Header Payload FCS Outer IP Outer UDP VXLAN Outer Ethernet Inner Ethernet Payload New FCS Instance ID 1 ReservedReservedFlags Rsvd Rsvd 8 Bytes 1 Byte Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789) Outer UDP Source Port = Hash of Inner Frame Headers (optional) 11. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Frame Format A Layer-2 Gateway bridges traffic to VLAN based on MAC DA A Layer-3 Gateway routes traffic to VLAN based on IP DA Original Ethernet Frame Outer MAC DA Outer MAC SA Outer 802.1Q Outer IP DA Outer IP SA Outer UDP VXLAN ID (24 bits) Inner MAC DA Inner MAC SA Optional Inner 802.1Q Original IP Payload CRC Allows for possible 16M segments IP header, allowing transport across any IP network Identifies packet as a VXLAN packet Transport VLAN Inner IP SA Inner IP DA 12. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Key Terminology VTEP NVE (Network Virtual Endpoint) VNI (VXLAN Network Identifier or VXLAN Segment ID) VXLAN Gateway Transit Remote VTEP Delivery Group (DG) BUM 13. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN VTEP VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point). Each VTEP has two interfaces - one to provide bridging function for local hosts, the other has an IP identification in the core network for VxLAN encapsulation/de- encapsulation. Local LAN Segment IP Interface End SystemEnd System VTEP Transport IP Network Local LAN Segment IP Interface End SystemEnd System VTEP 14. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Handling of Multi-Destination Traffic VTEP-1 End System A MAC-A IP-A VTEP-3 End System End System VTEP-2 End System B MAC-B IP-B Mcast Group IP Network VTEP 1 IP-1 VTEP 2 IP-2 VTEP3 IP-3 Since a control/signaling protocol has not been defined, emulation of Multi-Destination traffic (Broadcast, Multicast, Unknown Unicast) is handled through the VXLAN IP underlay through the use of segment control multicast groups 15. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Overview VTEP Discovery & Address Learning VTEP-2 End System B MAC-B IP-B VTEP 1 IP-1 MAC-1 VTEP 2 IP-2 MAC-2 VTEP3 IP-3 VTEP-1 VTEP- 3 End System End System Mcast Group 239.1.1.1 VXLAN VNID: 10 Outer S-IP: IP-1 Outer D-IP: 239.1.1.1 S-MAC: MAC-1 D-MAC: 00:01:5E:01:01:01 ARP Request for IP B Src MAC: MAC-A Dst MAC: FF:FF:FF:FF:FF:FF UDP 2 2 ARP Request for IP B Src MAC: MAC-A Dst MAC: FF:FF:FF:FF:FF:FF 3 3 ARP Request for IP B Src MAC: MAC-A Dst MAC: FF:FF:FF:FF:FF:FF ARP Response from IP B Src MAC: MAC-B Dst MAC: MAC-A 4 VXLAN VNID: 10 Outer S-IP: IP-2 Outer D-IP: IP-1 S-MAC: MAC-2 D-MAC: MAC-1 ARP Response from IP B Src MAC: MAC-B Dst MAC: MAC- A UDP 5 MAC Address VxLAN ID Remote VTEP MAC-A 10 IP-1 MAC Address VxLAN ID Remote VTEP MAC-A 10 IP-1 MAC Address VxLAN ID Remote VTEP MAC-B 10 IP-2 6 ARP Response from IP B Src MAC: MAC-B Dst MAC: MAC-A 7 End System A MAC-A IP-A ARP Request for IP B Src MAC: MAC-A Dst MAC: FF:FF:FF:FF:FF:FF 1 16. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Overview Unicast Forwarding Packet Flow Router-1 Host-A Host-B MAC-A IP-A: 10.1.1.100 MAC-B IP-B: 10.1.1.101 MAC-1 IP-1: 165.123.1.1 MAC-4 IP-4: 140.123.1.1 MAC-2 IP-2: 165.123.1.2 MAC-3 IP-3: 140.123.1.2 VXLAN VNID: 10 Outer S-IP: IP-1 Outer D-IP: IP-4 Outer S-MAC: MAC-1 Outer D-MAC: MAC-2 S-IP: IP-A D-IP: IP-B S-MAC: MAC-A D-MAC: MAC-B UDP UDP VXLAN VNID: 10 Outer S-IP: IP-1 Outer D-IP: IP-4 Outer S-MAC: MAC-3 Outer D-MAC: MAC-4 S-IP: IP-A D-IP: IP-B S-MAC: MAC-A D-MAC: MAC-B IP Network VXLAN VNID 10 (Tenant Blue) Router-2 VTEP-1 VTEP-2 S-IP: IP-A D-IP: IP-B S-MAC: MAC-A D-MAC: MAC-B 1 2 3 S-IP: IP-A D-IP: IP-B S-MAC: MAC-A D-MAC: MAC-B 5 Routed Based on Outer IP header 4 17. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 9000 Series VXLAN Support VXLAN is supported across the Nexus 9000 series platforms. The VXLAN Gateway functionality is supported across all form factors and line cards. Integrated routing functionality is only supported on ACI-enabled Modules Nexus 9500 SeriesNexus 9300 Series 18. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Supported Platforms Platform NX-OS version Minimum Recommended Nexus 9500 6.1.2I3.1.x 6.1.2.I3.1.x Nexus 9300 6.1.2I2.1.x 6.1.2.I2.1.x Nexus 3100 (3132/3172) 6.x Q2 CY14 Nexus 6000 7.0(0)N1x) Q3 CY14 Nexus 7000 with F3 7.0.x Q4 CY14 * There is no licensing cost for VXLAN Enhance Layer 3 19. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 9000 Series VXLAN Gateway VXLAN gateway bridges traffic between VXLAN segment and another physical / logical layer 2 domain (such as a VLAN) L3 Network VNI 1010 VNI 1020 VLAN 10 VLAN 20 VxLAN VTEP VxLAN VTEP (VxLAN Gateway) VxLAN VTEP VLAN ID VXLAN ID 10 1010 20 1020 20. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 9000 Series VXLAN Gateway VLAN ID VXLAN ID 10 1010 20 1020 VLAN to VXLAN Mapping VXLAN Forwarding Table Ethernet/ 802.1Q VXLAN EncapVTEP MAC Address VxLAN ID Remote VTEP AA:AA:AA:AA:AA:AA 1010 10.1.1.2 BB:BB:BB:BB:BB:BB 1020 10.1.1.3 feature nv overlay feature vn-segment-vlan-based interface et4/13 switchport switchport access vlan 10 no shut interface nve1 no shutdown source-interface loopback0 overlay-encapsulation vxlan member vni 1010 mcast-group 230.1.1.1 vlan 10 vn-segment 1010 switch# show nve vni Interface VNI Multicast-group VNI State ---------------- -------- --------------- --------- nve1 1010 230.1.1.1 up switch# show nve peers Interface Peer-IP VNI Up Time ------------- ---------------- -------- ------- nve1 10.1.1.2 1010 00:52:24 switch# The Nexus 9000 series supports VXLAN Gateway function, allowing VLANs to be bridged/mapped to VXLAN Segments and vice versa 21. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 9000 Series VXLAN Bridging VXLAN Bridging bridges traffic between VXLAN segments L3 Network VNI 1010 VNI 1020 VLAN 10 VLAN 20 VxLAN VTEP VxLAN VTEP (VxLAN Bridging) VxLAN VTEP VLAN ID VXLAN ID 10 1010 20 1020 22. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 9000 Series VXLAN Routed Mode VXLAN routed mode routes traffic between VXLAN segments and between VXLAN another physical / logical layer 2 domain (such as a VLAN) L3 Network VNI 1010 VNI 1020 VLAN 10VLAN 20 VxLAN VTEP (VxLAN Routed Mode) VxLAN VTEP VxLAN VTEP 23. VXLAN Design 24. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Designs High-level design options considered in this presentation are in the following areas: Routed Access + IP Mobility L2 extension across Pod / Multi-tenancy Datacenter Interconnect (DCI) 25. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Gateway Routed Access + IP Mobility VXLAN Enabled Hypervisor VTE P VXLAN Enabled Hypervisor VTEP VXLAN Gateway defined at access layer (leaf) Nexus 9000 Multicast needs to be enabled for VXLAN to work on the source interface Next hop of VTEP needs to be Layer 3 vPC needs peer gateway Only 1:1 mapping is allowed for VXLAN to VLAN Recommended N9K to be configured as STP root switch in each L2 network Link discovery protocols like CDP, LLDP will not discover neighbors on the remote VTEPs Virtual to physical migration (P2V) 26. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Forwarding Design Considerations VXLAN VTEP downstream of a Nexus 2000 FEX is not supported VTEP VXLAN VLAN When VXLAN is being routed the next hop for VXLAN encapsulated frames needs to be over an L3 interface Alternatively, all SVIs from a VXLAN Gateway must point to the same physical next hop [same VXLAN header MAC DA for all VXLAN encapsulated packets sent from the same physical port] 27. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Design with VXLAN Bridging only L2 Extension across Pods L3 Core Pod 1 Pod 2 VXLAN Overlay (VLAN Extension) Layer-2 VLAN Domain Layer-2 VLAN Domain IP GW IP GW VTEP (Layer-2 only) VTEP (Layer-2 only) L2 Link L3 Link 28. VXLAN Deployment Steps 29. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Sample Topology - VTEP Host A (VLAN 10) Host B (VLAN 10) L3 Transport Network (OSPF and IP PIM) Lpbk0: 100.100.100.1/32 Nexus 9000 VTEP-1 e1/1 Mcast grp: 230.1.1.1 RP: 10.1.1.1 Lpbk0: 100.100.100.2/32 .1 e2/1 20.1.1.0/30 .2 30.1.1.0/30 .1 e2/1 .2 Nexus 9000 VTEP-2 e1/1 30. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public VXLAN Sample topology with vPC Host A (VLAN 10) Host B (VLAN 10) Nexus 9000 VTEP-1 e1/1 L3 Transport Network (OSPF and IP PIM) Multicast group: 230.1.1.1 Loop backk0: 100.100.100.2/32.1 e2/1 20.1.1.0/30 .6 30.1.1.0/30 .1 e2/1 .2 Nexus 9000 VTEP-2 Nexus 9000 VTEP-3 .2 20.1.1.4/30 .5 e1/1 Loop back 0: 200.200.200.2/32 100.100.100.1/32 (Secondary) Loop back 0: 200.200.200.1/32 100.100.100.1/32 (Secondary) vPC e1/1 31. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Key Takeaways VXLAN Technology VXLAN is simple Keeps the attractive aspects of Layer 2 No re-addressing, simple configuration and deployment Integrates stability and scale of Layer 3 VXLAN is efficient Proper utilization of ECMP Optimal path between any two nodes VXLAN is scalable Can extend a bridged domain without extending the risks generally associated with Layer 2 and beyond 4K VLAN limit VXLAN Control Plane (Future) BGP and LISP 32. Cisco Nexus 7000 / 7700 Switch Architecture BRKARC-3470 33. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public What Is Nexus 7000? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7000 designed for general-purpose Data Center deployments, focused on 10G density plus 40G/100G I/O Modules Supervisor Engines Fabrics Chassis 34. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public What Is Nexus 7700? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7700 designed for SP and MSDC Data Center deployments, focused on high- density 40G/100G I/O Modules Supervisor Engine Fabrics Chassis 35. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 36. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 7700 Chassis Family Front Rear 26RU N77-C7718 Nexus 7718 Front Rear 14RU N77-C7710 Nexus 7710 Front Rear 9RU N77-C7706 Nexus 7706 NX-OS 6.2(6) and later NX-OS 6.2(2) and later NX-OS 6.2(2) and later Front Back Front Back Front Back 37. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 38. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Next generation supervisors providing control plane and management functions Connects to fabric via 1G inband interface Interfaces with I/O modules via 1G switched EOBC Second-generation dedicated central arbiter ASIC Controls access to fabric bandwidth via dedicated arbitration path to I/O modules Supervisor Engine 2 / 2E Console Port Management Ethernet N7K-SUP2/N7K-SUP2E USB Host Ports ID and Status LEDs Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700) Base performance High performance One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM USB Log Flash USB Expansion Flash N77-SUP2E ID and Status LEDs Console Port Management Ethernet USB Expansion Flash 39. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public 10G / 40G / 100G M2 I/O modules Share common hardware architecture Two integrated forwarding engines (120Mpps) Support for XL forwarding tables (licensed) Distributed L3 multicast replication 802.1AE LinkSec on all ports Supports Nexus 2000 (FEX) connections (10G) N7K-M224XP-23L Nexus 7000 M2 I/O Modules N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L Supported in NX-OS release 6.1(1) and later N7K-M206FQ-23L N7K-M202CF-22L Module Port Density Optics Bandwidth M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G M2 100G 2 x 100G CFP 200G 40. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 7000 / 7700 F2E I/O Modules N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E 7000: Supported in NX-OS release 6.1(2) and later 7700: Supported in NX-OS release 6.2(2) and later N7K-F248XP-25E N7K-F248XT-25E 48-port 1G/10G with SFP/SFP+ transceivers 480G full-duplex fabric connectivity System-on-chip (SoC) forwarding engine design 12 independent SoC ASICs Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) Interoperability with M1/M2, in Layer 2 mode on Nexus 7000 Proxy routing for inter-VLAN/L3 traffic LinkSec support* Last 8 ports (SFP+) All 48 ports (Copper) * Roadmap item N77-F248XP-23E 41. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public DC EDGELEAF SPINE Integrated AND rich for Core, Spine, Leaf, DCI, SAN Deployments MOST COMPREHENSIVE Multi-tenancy and Virtualization Capable Hardware READY FOR Energy Efficient ENVIRONMENTAL UNPRECEDENTED Investment Protection on Nexus 7000 Nexus 7700 24 port 40GE Nexus 7700 12 port 100GE Q4 CY13 Nexus 7000 12 port 40GE Nexus 7000 6 port 100GE Q4 CY13 Q4 CY13 Q1 CY14 F3-Series Modules Nexus 7000 / 7700 F3 I/O Modules 42. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 7700 F3 12-Port 100G Module Architecture Front Panel Ports (CPAK) To Fabric Modules To Central Arbiters Arbitration Aggregator 1 X 100G SoC 2 2 1 X 100G SoC 3 3 1 X 100G SoC 4 4 1 X 100G SoC 5 5 1 X 100G SoC 6 6 1 X 100G SoC 7 1 X 100G SoC 8 1 X 100G SoC 9 1 X 100G SoC 10 1 X 100G SoC 11 Fabric ASIC Fabric ASIC 7 8 9 10 11 1 X 100G SoC 12 12 1 X 100G SoC 1 1 FSA CPU EOBC LC Inband 1G switch x 12 to FSA CPUto ARB x 12 x 6 43. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 44. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public M-Series Forwarding Engine Hardware Two hardware forwarding engines integrated on every M2 I/O module 120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning 120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4 60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir) MPLS/VPLS/EoMPLS OTV RACL/VACL/PACL QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard IGMP snooping Ingress and egress NetFlow (full and sampled)Hardware Table M-Series Modules without Scale License M-Series Modules with Scale License MAC Address Table 128K 128K FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6 Classification TCAM (ACL/QoS) 64K 128K NetFlow Table 1M 1M 45. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public F3 Forwarding Engine Hardware Each SoC forwarding engine services: 8 front-panel 10G ports 2 front-panel 40G ports 1 front-panel 100G port 148Mpps per SoC Layer 2 bridging with hardware MAC learning 148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*) RACL/VACL/PACL QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard IGMP snooping FabricPath forwarding Overlay Transport Virtualization (OTV) MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE* Ingress/egress* sampled NetFlowHardware Table Per F3 SoC Per F3 Module MAC Address Table 64K 384K/768K** FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6 Classification TCAM (ACL/QoS) 16K 96K/192K** ** Assumes specific configuration to scale SoC resources * Roadmap items 46. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 47. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Crossbar Switch Fabric Modules Provide interconnection of I/O modules Each installed fabric increases available per-payload slot bandwidth Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC Different I/O modules leverage different amount of available fabric bandwidth Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ N7K-C7018-FAB-2 N7K-C7010-FAB-2 N7K-C7009-FAB-2 Fabric Module Supported Chassis Per-fabric module bandwidth Max fabric modules Total bandwidth per slot Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot N77-C7718-FAB-2 N77-C7710-FAB-2 N77-C7706-FAB-2 48. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public 110G (2 x 55G) Ingress Module Egress Module Multistage Crossbar Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric Stages 1 and 3 on I/O modules Stage 2 on fabric modules 1st stage Egress Module 2nd stage Ingress Module 3rd stage Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASICFabric ASIC Fabric ASIC Fabric Modules Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 1 Fabric ASIC 2 3 4 5 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 6 Fabric ASIC 1.32T 1st stage 3rd stage 550G 110G (2 x 55G) 1 Fabric ASIC 2 3 4 5Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Nexus 7000 Nexus 7700 49. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public 220Gbps440Gbps660Gbps880Gbps1100Gbps1320Gbps Local Fab2 #1 (480G) Local Fab2 #1 (960G) Local Fab2 #1 (1.2T) Fab2 #2 Fab2 #2 Fab2 #2 I/O Module Capacity Nexus 7700 One fabric: Any port can pass traffic to any other port in VDC Three fabrics: 480G F2E/F3 10G module has maximum bandwidth Five fabrics: 960G F3 40G module has maximum bandwidth Six fabrics: 1.2T F3 100G module has maximum bandwidth per slot bandwidth Fabric 2 Modules 1 Fabric 2 ASICs 2 Fabric 2 ASICs 3 Fabric 2 ASICs 4 Fabric 2 ASICs 5 Fabric 2 ASICs 6 Fabric 2 ASICs 50. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Fabric, VOQ, and Arbitration Crossbar fabric Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules Virtual Output Queues (VOQs) Provide buffering and queuing for ingress- buffered switch architecture Central arbitration Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch 51. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 52. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Buffering, Queuing, and Scheduling Buffering storing packets in memory Needed to absorb bursts, manage congestion Queuing buffering packets according to traffic class Provides dedicated buffer for packets of different priority Scheduling controlling the order of transmission of buffered packets Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal priority Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define buffering, queuing, and scheduling behavior Default queuing and network-QoS policies always in effect in absence of any user configuration 53. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing NetFlow Conclusion 54. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Full vs. Sampled NetFlow NetFlow collects full or sampled flow data Full NetFlow: Accounts for every packet of every flow on interface Available on M-Series modules only Flow data collection up to capacity of hardware NetFlow table Sampled NetFlow: Accounts for M in N packets on interface Available on both M2 (ingress/egress) and F2E/F3 (ingress only) M2: Flow data collection up to capacity of hardware NetFlow table F2E/F3: Flow data collection for up to ~1000pps per module F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex 55. 2014 Cisco and/or its affiliates. All rights reserved.BRKDCT-1301 Cisco Public Nexus 7000 / Nexus 7700 Architecture Summary I/O Modules Supervisor Engines Fabrics Chassis