Switch Hardware Architecture

8/12/2019 Switch Hardware Architecture

1/65

BRKRST-3069

Cisco Switching Hardware ArchWhat makes a Cisco Switch

www.ciscolivevirtual.com
http://www.ciscolivevirtual.com/http://www.ciscolivevirtual.com/


2/65

2012 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKRST-3069

Agenda

Overview

Concept

System Design

Mechanical / PhysicalDesign

Buffer Design

Forwarding Design

ASIC Engineering

Hardware Engine

Software Enginee


3/65

Overview


4/65


Timeline

ASIC

Requirements Plan Micro Architecture Implementation Final Netli

Hardware

HW Design

Mechanical

Electrical

Manufacturing

Mechanical Drawing

PCB LayoutMDVT

EDVTBOM

P0 P1Fab OutDetailed Design

Product Requirements Document

Software

SW Functional Spec SW Design Spec Unit Test Plan Unit Inte

Software Test

Master Test Plans Functional Test Plans Automation Re


5/65 2012 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKRST-3069

Nexus 7000 and F2 Modules


6/65

Concept



Concept

Vision

Market

Cost

Time to Market

Differentiation

Innovation

Technology

Life Cycle

How Big?

How many ports?

Fixed vs Modular

Backward Compa

What customer problem will the product solve?



Ciscos End-to-End Data Centre Switchin

platform; providing solutions for 10G, 40and 100G for Access, Aggregation, and C

Consolidate IP, Storage, and IPC network

onto a single Ethernet fabric and deliverinnovative features and services that provalue to our customers.

Nexus 7000 Vision (circa 2007)



DC Evolutionary Innovation

10G Aggregation

10GbE Access

10GbE Aggregation

Unified Fabric

10G Access40 / 100G Aggregation

Unified Fabric

FCS

DC3

80G Slot

2009 Phase 1

Terabit Slot

2011 Phase 2 Terabit Slot

201

Cisco Interna



F2 Series

48 Ports 1/10G

Line Rate 64 Bytes

Low Latency

L2MP, TRILL, FEX,FCoE, L3 Forwarding

Optimise for Data Centre

IPv4 & IPv6Equal Performance

Cost target

High Level Goals


11/65

System Design


12/65


Many Factors to Weigh

Standards requirements

Market requirements

Designability

Silicon technology

Processor technology

Manufacturability

Time to market

Flexibility

Budget

Modular / Fixed

Applicable to any Switch / Router Design


13/65


Many Factors to Weigh

Buffering

No packet drop

Throughput

Port count

Modular

No single point of failure

In-order delivery

Future protocol compatibility

Modular

Restartable (includingactive state handling)

Non-disruptive code loactivation

No single point of failu

Scaleable

Unit Testable

Future protocol comp

Baseline Data Centre Switch RequirementsData Plane Control plane


14/65

Mechanical / Physical Design


15/65


Mechanical Design

Nexus 7010 Rear Nexus 7010 Front

N7K-AC-6.0kW

Power Supply

Fabric

N7K-C7010-FAB-1

48

N7

Nexus 7010 Rear


16/65


Industrial Design / Usability

Eje


17/65


Industrial Design / Usability


18/65

Buffer Design


19/65


One plus one does not equal Two

SwitchA

SwitchB

10 Links @ 1Gbps Each

Bandwidth = 10Gbps

Flow Bandwidth = 1Gbps

Serialisation Delay = 20uS

1 Link @ 10Gbps

Bandwidth = 10Gb

Flow Bandwidth =

Serialisation Dela

SwitchA


20/65


Single ASIC

Input3

Input2

Input1

Scalability limited by memorybandwidth/size

Typically optimised for fixedconfiguration

Cost effective with

small port counts

Often used asbuilding block


21/65


Switch Architecture

Mesh

Clos / Fat Tr

Crossbar


22/65


Complete SystemPull Fabric

Fabric

Arbiter

Request

Grant

Credit

Superframes Superframes

DWRR

SP

WRED

WRED

WRED

DWRR

SP

WRED

WRED

WRED

1

2

3 4

5 6

7

WRED

WRED

WRED

Ingress Egress


23/65

Forwarding Design


24/65


High Level View of Forwarding

L2 Table

TableLookups

Classification

ForwardinDecision

ParsePacket

L3 Table


25/65


How fast?

GT/s Serdes

(Gbps)

Encoding

PCI express v1 2.5 2.525 8b/10b

PCI express v2 5 5G 8b/10b

PCI express v3 8 7.99 128b/130b

10G Ethernet 10.3125 64b/66b

10G Ethernet = 14.88Mp

67.2ns to receive a pack

100G Ethernet = 148.8M6.72 ns to receive a pac

DDR3 Latency ~10ns

SRAM Latency 1 cycle


26/65


10G Ethernet Forwarding Rate

T bl L k


27/65


Table LookupsCAMs, HASH Tables and *Tries

010010101

010010XX2

01001XX03

01001XXX4

Input Key

Result

CAM Hash Table


28/65


CAMs

011010101

011010112

010011103

011011004

01001110

Hit!

3 Result

Content AddressableMemory

010010101

010010XX2

01001XX03

01001XXX4

01001110

Hit #1

2

4

3

01001101

01001000

Hit #3

Hit #2

Lkup #1

Lkup #2

Lkup #3

Result #1

Result #2

Result #3

Ternary ContentAddressable Memor

Storing 1 bit in TCAM takes 10


29/65


Hash Tables

1 bit in SRAM takes

1 bit in DRAM takes

Input MAC Address Pages

0000.c000.0001

Mathematical Functional

produce value between

0 and Page Size

Pag

Compare if value in each page

matches input value


30/65


Tries

Many different *tries

Bitwise Trie

Balanced Trie

Patricia Trie

Fixed or Variable Stride Tries

Store information in eachleaf or pointer to table withinformation in it


31/65


IPv4 Unicast FIB

VRF / Prefix / Mask / Paths / Offset

1 / 10.1.2.0 / 24 / 4 / 11 / 10.1.3.0 / 24 / 1 / 5

3 / 10.1.2.0 / 24 / 2 / 9

3 / 10.1.3.0 / 24 / 2 / 9

L3 Table: Design 1

Rewrite

ADJ 1 - Rewrite SR

ADJ 2 - Rewrite SR

ADJ 3 - Rewrite SR

ADJ 4 - Rewrite SR

ADJ 5 - Rewrite SR

ADJ 6 - Rewrite SR

ADJ 7 - Rewrite SR

ADJ 8 - Rewrite SR

ADJ 9 - Rewrite SR

ADJ 10 - Rewrite S

Software View

H

A

S

H


32/65


IPv4/v6 Unicast FIB

VPN / Prefix / Mask / Paths / Offset

1 / 10.1.2.0 / 24 / 4 / 11 / 10.1.3.0 / 24 / 1 / 5

3 / 10.1.2.0 / 24 / 2 / 6

3 / 10.1.3.0 / 24 / 2 / 6

Path Table

Path 1

Path 2

Path 3

Path 4

Path 1

Path 1

Path 2

L3 Table: Design 2

Rewrite

ADJ 1 - Rewrite SR

ADJ 2 - Rewrite SR

ADJ 3 - Rewrite SR

ADJ 4 - Rewrite SR

ADJ 5 - Rewrite SR

ADJ 6 - Rewrite SR

ADJ 7 - Rewrite SR

ADJ 8 - Rewrite SR

ADJ 9 - Rewrite SR

ADJ 10 - Rewrite S

HA

S

H

Software View


33/65


L2 Table / Host Table / FIB

Hash tables take less space than TCAMs and

Instead of placing /32 or /128 entries for hostentries into the FIB, place them into the hash t

Common for the L2 table and the Host table toshare the same memory

Allows for the FIB Table to be smaller since it dnot need to contain single path /32 and /128 e

Common Optimisation


34/65


Forwarding Design

L2 Table

L3 Table

Ingress

Security ACLs

Ingress QoS

ACL

Adjacency

Table

EgressSecurity ACLs

Egress QoS

ACL

L2 Table

L3 Table (x2)

Ingress Security

ACLs

Ingress QoS ACL

Adjacency

Table

Egress

Security ACLs

Egress QoS

ACL

VPN

CAMInput / Output

Policing

Parse

Packet

Input / Output

PolicingFwd Decision

Parse

Packet

Design 1

Design 2


35/65


Forwarding Design

L2 Table

L3 Table (x2)

Ingress SecurityACLs

Ingress QoS ACL

AdjacencyTable

Egress

Security ACLs

Egress QoS

ACL

VPN

Table

Input Policing

Parse

Packet

L2 Table L3 Table (x2)

Ingress Security

ACLs

Ingress QoS ACL

Adjacency

Table

Egress

Security ACLs

Egress QoS

ACL

VPN

Table

Fwd

Decision

Parse

Packet

Input / OutputPolicing

Output Policing

Design 2

Design 3


36/65


References

Network Algorithmics,: An Interdisciplinary

Approach to Designing Fast Networked DevicGeorge Varghese

Art of Computer Programming Vol 1-4, DonaldKnuth

Introduction to Algorithms, Third EditionThomas H. Cormen, Charles E. Leiserson, RoL. Rivest and Clifford Stein

IEEE SIGCOMM Papers


37/65

ASIC Engineering

ASIC FPGA


38/65


ASICs vs FPGAs

ASIC - Application SpecificIntegrated Circuit

A finished IC which is built to theexact specification & functionalityof the customer

Can make optimal use of theunderlying silicon circuits

Low part cost, High upfront

investment Significant development time

FPGA (EPLD) Field ProgramGate Array

An IC that can be configuthe required functionality ainstalled into a target syst

Flexibility vs. sub-optimal underlying silicon circuits

Higher part cost

Shorter development time

Main players: Xilinx, Altera

CMOS


39/65


CMOS

http://en.wikipedia.org/wiki/Semicondu

Feature size

This dimension is what

Moores Law is all about!

in

out

VDDVSS

out

p+ p+n-well

n+ n+

in

1.6 X Increase in usable gates betw

G t


40/65


Gates

module nand2(a,b,c)

input a,b;

ouput c;

begin

c


41/65


Why is die size important?

300mm

With same number of defects per wafer,

smaller Die size results in higher yield per wafer

Defe

Silicon Wafer

Die

I t t d Ci it P d ti


42/65


Integrated Circuit Production

Floor PlanOverall Block and

Function Placement PlacementSpecific Gate

Placement

RouteLayout physical

interconnection

GDSIIOne file per layer

(photomask)

Foundry

ProductionMetal layers on

WaferDevice Test

Test Dies on Wafer PackagingCut wafer into dies

Dies into IC packages

RTLRegister Transfer

Language

Verilog, VHDL

SynthesisTurn RTL into Gates and

Logical Connections

NetlistGates and Logical

Interconnections

Integrated Circuit Production


43/65


Integrated Circuit ProductionRTL

Register Transfer

Language

Verilog, VDHL

SynthesisTurn RTL into Gates and

Logical Connections

NetlistGates and Logical

Interconnections

ASIC Customer- Cisco

Floor PlanOverall Block and

Function Placement PlacementSpecific Gate

Placement

RouteLayout physical

interconnection

GDSIIOne file per layer

(photomask)

Foundry

ProductionMetal layers on

WaferDevice Test

Test Dies on Wafer PackagingCut wafer into dies

Dies into IC packages

S-

G

ASIC Design Process


44/65


ASIC Design Process

Requirements

Planning

Micro

Architecture

Implementation

Final Netlist

Power On

Select vendor,

process, package

Architecture, HW, SW,Marketing sign-off

Requirements Complete

ASIC Commit

Design Review

Final Netlist Handoff

Mask order

Floorplan Netlist

RTL Release

Prelim Netlist

DV Review

12-26 Weeks

~52 Weeks ~12 Weeks~12 Weeks

F2 ASIC Clipper


45/65


F2 ASIC - Clipper

Technology

Die Size 1

Total SRAM

Total eDRAM

Total TCAM

Register Array

Logic Gates Signal Pin

Package IO

Memory and Packet Corruption Prot


46/65


Memory and Packet Corruption Prot

No ECC or Parityno way to determine if a

software or hardware problem

Paritywill detect single bit errors

ECCwill detect 2 bit errors, and correct sing

Parity and ECC apply to a word (32 or 64 bits)

CRCDetect if a set of bytes (normally a pachas been corrupted

ASIC Packaging


47/65


ASIC Packaging

Electrical parasitics of thechip package are critical

Impacts electrical propertiesof high-speed signals

Manufacturing tolerancesconstrain minimum ball pitch

Limit to number of availablesignal I/O pinsLevel-1 Interconnect

Die-to-Package

Package Substrate

Power Planes

(FR4/Ceramic)

Silicon Die

Underfill


48/65

Hardware Engineering

F2 Block Diagram


49/65


F2 Block Diagram

Lightning

Central Arbiter

Sacramento

To Spine Cards

Clipper

EDC

SFP+

SFP+

SFP+

SFP+

Clipper ClippeClipper ClipperClipper ClipperClipper ClipperClipper

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

SFP+

EDC

SFP+

SFP+

SFP+

Thermal Modelling


50/65


Thermal ModellingTemperature ConComponent Case Temperatures

Electrical / Mechanical Layout


51/65


Electrical / Mechanical Layout

EDVT(Electronic Design Validation Test)


52/65


(Electronic Design Validation Test)

All tests performed using offline diagnosticsand again with NXOS

On-board power supplies have voltagesmargined to +5% & -5%

Temperature testing occurs while

Soaking for 12 hours at 55o C and -5o C

Ramping between extremes at 1o C perminute

Power cycle testing occurs during 12-hoursoak

RDT (Reliability Demonstration Test)1


53/65


RDT (Reliability Demonstration Test)1

The Reliability Demonstration Test (RDT) isCiscos approach to verifying the statedreliability of a product prior to production

release.

The reliability to be demonstrated is theproducts MTBF (Mean Time BetweenFailure).

RDT replicates the end user operating

environment and application throughaccelerated test time. It is expected that allhardware features are exercised in RDT.

All new products including systems andboards are subject to RDT.

Power Consumption


54/65


Power Consumption

Skew Parts

Data Sheet

Typical 340W

Maximum 400W

Generic Online Diagnostics


55/65


Generic Online Diagnostics

Generic Online Diagnostics provide a diagnostic framework for d

hardware faults and verifying the health of hardware component

throughout the chassis.

Hardware Components (ASICs)

Interfaces (Ethernet, SFP+, etc)

Connecters (loose connectors, bent pins, etc)

Memory Failure (Failure over time)

Solder Joints

Diagnostics run during system Boot-Up, after OIR, On-Demand

CLI, or as Health Checks in the background.

Problem Areas:


56/65

Software Engineering

NXOS Architecture


57/65


NXOS ArchitectureLayer-2 Protocols Storage ProtocolsLayer-3 Protocols

Interface Management

Chassis Management

Kernel

Sysmgr,PSS&

MTS

SNMP,XML,CLIManagement

Chip/Driver Infrastructure

VLAN mgr

STP

OSPF

BGP

EIGRP

GLBP

HSRP

VRRP

VSANs

ZoningFCIP

FSPF

IVR

UDLD

CDP

802.1XIGMP snp

LACP PIMCTS SNMP

Other Servic

Future Serv

Possibilities

Protocol Stack (IPv4 / IPv6 / L2)

Multi-threaded Real-Tim


58/65


Scalability with SMP and multi-core CPUs

Faster Route Re-convergence

Lower mean-time-to-recovery

Real-Time preemptive sche

System operational when C

ModularityMost of the features are conditional

Can be enabled/disabled independently

Maximises efficiency

Minimises resources utilisation

Separation Control Plane andData Plane

No software forwarding feature

Fully distributed hardware forwarding

Line Card OffOffload to line card C

Scales with # of line c

Optimal hardware pro

Software Engineering


59/65


g g

} mfib _hw_o if_t;

MDT

Idx1

Idx2

IPv4 (S, G) Database

Pltfm Data

Table Ptr.MFIB Context

(S, G)

Prefix

rpfif/df

Pltfm Data

hw_idx[]

md_adj[..]

(S, G)

Prefix

rpfif/df

Pltfm Data

hw_idx[]md_adj[.]

(S, G)

Prefix

Pltfm Datahw_idx[]

md_adj[..]

FIB DRAM

(S, G)

OIF List

Pltfm Data

MET1 Ptr

ADJ RAM

MD Adj

OIF1 Adj

RIT RAM

ccc=7

OIF List

Pltfm Data

MET1 Ptr

OIF 2 Adj.

IPv6 (S,G) Database

SW Functional Spec SW Design Spec Unit Test Plan Unit In

Design Review


60/65


g

Development Test


61/65


p

Testing of completed integ

feature Test for interactions with o

features and functions

Test for interoperability wit

Cisco and 3rdparty device

Build scripts to automate tso is repeatable on futurereleases

Master Test Plans Functional Test Plans Automation Regress

First Customer Ship


62/65


pASIC

Requirements Plan Micro Architecure Implementation Final Netlis

Software

SW Functional Spec SW Design Spec Unit Test Plan Unit Inte

Software Test

Master Test Plans Functional Test Plans Automation Reg

HardwareHW Design

Mechanical

Electrical

Manufacturing

Mechanical Drawing

PCB LayoutMDVT

EDVTBOM

P0 P1Fab OutDetailed Design


63/65

Q & A

Complete Your Online SessionEvaluation


64/65


Evaluation

Complete your session evaluation:

Directly from your mobile device by visitingwww.ciscoliveaustralia.com/mobileand loginby entering your username and password

Visit one of the Cisco Live internetstations located throughout the venue

Open a browser on your own computerto access the Cisco Live onsite portal Dont forget to activate

Virtual account for acce

materials, communities

live activities throughou

account at any internet

www.ciscolivevirtual.co
http://www.ciscoliveaustralia.com/mobilehttp://www.ciscolivevirtual.com/http://www.ciscolivevirtual.com/http://www.ciscoliveaustralia.com/mobile


65/65


Switch Hardware Architecture

Documents

Transcript of Switch Hardware Architecture