Layer 3 Forwarding and Troubleshooting...

Layer 3 Forwarding and Troubleshooting Deep

Dive on Nexus 9000/3000

Ambrish Mehta – Principal Engineer (INSBU Engineering)

Swami Narayanan – Principal Engineer (INSBU Engineering)

Session Abstract

• Closer look at various Layer 3 forwarding options available on Nexus 95xx/93xx/31xx/32xx data center switches.

• Details on Layer 3 forwarding in multi chip and single chip hardware architecture.

• Flexibility offered by Forwarding Engine(ASIC) and how NX-OS is utilising it to meet different deployment needs.

• Empower Network Engineer/Architect to identify ideal routing mode for network deployment.

Related Sessions

BRKDCT-3640 - Nexus 9000 architecture

Friday 2 to 4 PM

Room # 211

Glossary of TermsNFE Network Forwarding Engine (Trident2 ASIC)

NFE2 Network Forwarding Engine 2 (Tomahawk ASIC)

DMOD Destination Module (ASIC Instance)

DPORT Destination Port (Port within ASIC instance)

UFT Unified Forwarding Table

LC Line card Module

FM Fabric Module

LPM Longest Prefix Match (e.g IPv4 : /31 to /0, IPv6 /127 to /0)

ALPM Algorithmic Longest Prefix Match

ECMP Equal Cost Multipath

• Introduction to 95xx/93xx/31xx/32xx

• Layer 3 Forwarding Architecture

• NFE/NFE2 Forwarding Architecture

• NX-OS Routing Modes and Deployment Scenarios

• Walk Through of Commonly used Layer 3 Commands

• Q & A

Agenda

Introduction to 95xx/93xx/31xx/32xx

Nexus 95xx/93xx Product Family

Nexus 9500 Series Switches

Nexus9500 Series

Nexus 9300 Series Switches

Nexus9504/Nexus9508/Nexus9516N9K-C9332PQ N9K-C9372PX N9K-C9372TX

N9K-C9396PX N9K-C93128TX

Nexus9300 Series

Nexus 31xx/32xx Product Family

Nexus 31xx Series Switches

Nexus 31xx

Nexus 32xx Series Switches

N3K-3132Q N3K-3164Q N3K-3232-C N3K-3264-C

Nexus 32xx

Layer 3 Forwarding Architecture

Layer 3 Forwarding Architecture Categories

• Modular Multi-Chip

• Modular switches (95xx) belongs to this category.

• LC and FM Chip operates independently.

• Fixed Multi-Chip

• 3164Q belongs to this category

• Virtual/Pseudo LC and FM operates independently.

• Fixed Single-Chip

• 93xx, 3132Q, 3232-C and 3264-C belongs to this category

• Single Chip is making all forwarding decisions.

Modular Multi-Chip: LC and FM Connectivity

• NFE: Network Forwarding Engine (Forwarding ASIC)

• Each of the 3 NFEs on the line card has 12 x 40 Gbps links to fabric modules

• The above diagram shows the line card fabric connectivity in a Nexus 9508 chassis as an example

• LC and FM are connected via HiGig Trunk (Port-Channel)

• Each Front Panel port in the switch has unique identifier (DMOD/DPORT)

NFE NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 2

NFE NFE

Fabric 3

NFE NFE

Fabric 4

NFE NFE

Fabric 5

NFE NFE

Fabric 6

NFE

HiGig Trunk (Port-Channel)

36-port 40 Gbps QSFP+

Line Card/IO moduleDMOD 1 DMOD 2 DMOD 3

DMOD 100 DMOD 100 DMOD 100 DMOD 100 DMOD 100 DMOD 100

Interface DMOD DPORT

Eth1/1 1 1

Eth1/36 3 12

Fabric

Module

Modular Multi-Chip : Layer 3 Building Blocks

• Separate but identical NFE is present on LC and FM

• We can make use of both to program routes and get higher scale

• Treat LC and FM NFEs as independent forwarding entity

• Treat IPv4 Host, IPv4 LPM, IPv6 Host and IPv6 LPM as independent entity

• Identify programming location of prefixes blocks based on the Configured Routing Mode.

IPv4 Host IPv4 LPM

IPv6 Host IPv6 LPM

Prefix Blocks

Modular Multi-Chip : Layer 3 Building Blocks

IPv4 Host IPv4 LPMIPv6 Host IPv6 LPM

Fabric Module

Line Card

NFE

NFE

IPv4 Host IPv6 Host

IPv6 LPMIPv4 LPM

Fabric Module

Line Card

NFE

NFE

IPv4 Host

IPv6 Host

IPv4 LPM

Fabric Module

Line Card

IPv6 LPM

NX-OS SoftwareOption

1Option

2

Option

3

NFE

NFEIPv4 Host

IPv6 Host IPv6 LPM

IPv4 LPM

Modular Multi-Chip : Hierarchical Routing

NFE NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module


… Fabric Modules

LC and FM both have same NFE

Use 2 Stage Forwarding Decision to efficiently utilise LC and FM tables (Higher Route Prefix Scale)

LC NFE handles 1 set of forwarding lookup, FM NFE handles other set

LC NFE forwarding decision HIT, Egress Point of Exit (DMOD, DPORT) derived

Packet forwarded to Egress Interface directly.

Incase Egress interface in different NFE, then goes through FM NFE (only Hybrid Switching and no

forwarding lookup)

Hierarchical Routing – LC Lookup

NFE NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

…Fabric ModulesHybrid Switching

LC NFE HIT ->

Egress DMOD,

DPORTDMOD 3DMOD 1 DMOD 2

3 12 Data Packet

HiGig

Header

Eth1/1 Eth1/36

• LC NFE – Catch All Entry to punt packet to FM NFE for lookup

• Special DMOD (DMOD 100) in HiGiG Header. FM NFE does forwarding lookup if DMOD is 100

• FM NFE Lookup HIT – Packet Forwarded to Egress Interface (DMOD, DPORT)

• FM NFE Lookup NO HIT – Packet punted from FM NFE to SUP for ICMP Unreachable (Rate policed)

Hierarchical Routing – FM Lookup

NFE NFE NFE

NFE

DMOD 100

NFE NFE

DMOD 100

NFE

Line Card/IO module

…Fabric Modules

LC NFE Miss, Hit

Catch All -> Punt to

FM (DMOD 100)

FM NFE Hit =>

Egress DMOD,

DPORT

100 - Data Packet

HiGig

Header

3 12 Data Packet

HiGig

Header

DMOD 1 DMOD 3DMOD 2

Eth1/1 Eth1/36

• Single Stage Forwarding Decision (only LC NFE)

• Fabric NFE used for Hybrid Switching only

• Scale limited to Single Chip mode setting

• No HIT, packets punted to SUP for ICMP Unreachable (Rate policed)

Non Hierarchical Routing

NFE NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

… Fabric Modules

Hybrid

Switching

LC NFE HIT -> Egress

DMOD, DPORT

Eth1/1 Eth1/36

Fixed Multi-Chip : Pseudo LC and FM Connectivity

Same as 95XX Modular Chip Forwarding Architecture

Use 2 Stage Forwarding Decision to efficiently utilise LC and FM tables (Higher Route Prefix Scale)

LC NFE handles 1 set of forwarding lookup, FM NFE handles other set

All modes supported in Fixed Multi-Chip operates same as Modular Multi Chip

NFE NFE NFE

NFE NFE

Pseudo LC


Pseudo FM

NFE

HiGig Trunk (Port-Channel)

DMOD 1 DMOD 3DMOD 2

DMOD 100

DMOD 4

Fixed Single-chip

• All IPv4 and IPv6 host and LPM routes are programmed on this Single NFE2

• Single ASIC needs to do all forwarding decision

NFE2

32 x 100 Gbps

IPv4: Host and LPM Routes

IPv6: Host and LPM Routes

N3k-3232-C

Flexibility offered by NFE/NFE2

Unified Forwarding Table Bank Allocation

Bank 0-1

Dedicated

L2 Bank

Banks 2-3

Configurable

for

MAC/Hosts/

LPM

Bank 6-9

Dedicated

L3 Hosts

(V4 + V6

combined)

Banks 4-5

Configurable

for

MAC/Hosts/

LPM

16K LPM

Table for V4

and V6

LPM routes

UFT Mode 2

Bank 0-1

Dedicated

L2 Bank

Banks 2-3

Carved for

Layer 2

MAC

Bank 6-9

Dedicated

L3 Hosts

(V4 + V6

combined)

Banks 4-5

Carved for

Layer 3 V4

& V6 Hosts

16K LPM

Table for V4

and V6

LPM routes

UFT Mode 3

Bank 0-1

Dedicated

L2 Bank

Bank 2

Carved for

L2

Bank 6-9

Dedicated

L3 Hosts

Hosts(V4 +

V6

combined)

Banks 4-5

Carved for

Layer 3 V4

+ V6 Hosts

16K LPM

Table for V4

and V6

LPM routes

Bank 3

Carved for

L3 V4 + V6

Hosts

NFE TCAM LPM Table

IPv4/IPv6

(<=/64)

IPv6

(/65 to /127)

IPv6 /65 to /127 prefix length carving configuration

N9k(config)# hardware profile ipv6 lpm-entries maximum <256-3072>

Warning: The command will take effect after next reload.

N9k(config)#

16K

LPM Type # TCAM

Entries

LPM Scale

V4 * 1 12K/4K

V6 (<=/64) * 2 6K/2K

V6 (/65 to /127) 4 1K/3K**

* Shared TCAM space between IPv4 LPM and IPv6 LPM <=/64

** Default TCAM allocation allows1K prefixes IPv6 /65 to /127

1

1 1

1 1 1 1

UFT Mode 4

Bank 0-1

Dedicated

L2 Bank

Banks 2-3

Carved for

storing the

V4/V6 LPM

prefixes

Bank 6-9

Dedicated

L3 Hosts

(V4 + V6

combined)

Banks 4-5

Carved for

storing the

V4/V6 LPM

prefixes

16K LPM

Table

(Indexing to

Bank2-5)

ALPM and IPv6 Modes

• Algorithmic LPM - Repurposed UFT table to achieve higher LPM scale for IPv4 and IPv6

• Supports two modes for IPv6 LPM (64B and 128B)

• IPv6 mode setting at NFE level [not individual TCAM banks]

128B mode : All LPM prefix length uses128 bit width entries

64B mode : No /65 to /127 route can be programmed on NFE

• 128B Mode : V6 LPM Scale Tunable with configuration

IPv6 ALPM ModesALPM Mode NFE NFE2

64B V4 LPM : 128K

V6 LPM : 80K

V4 LPM : 128K

V6 LPM : 80K

128B V4 LPM : 128K/64K

V6 LPM : 16K/20K *

V4 LPM : 128K/64K

V6 LPM : 10K/15K *

* Achieved with carve-value configuration knob

N9k(config)# hardware profile ipv6 alpm carve-value <1024-3072>


N9k(config)#

Route Lookup and Rewrite

• Unicast Route Forwarding has 2 components

Route Lookup

Rewrite packet for next hop

• Rewrite information contains (Next Hop MAC, Egress Interface)

• ARP/Neighbor Discovery provides the information

• NFE stores these details in Rewrite table (Egress Next Hop Table)

Egress NH Table Table Size – 48k/32k (NFE/NFE2)

UFT Scale

Mode L2 L3 Host * LPM *

0 288K 16K 16k

1 224k 56k 16k

2 160k 144k 16k

3 96k 208k 16k

4 32k 16k 128k

* Number of IPv4 Entries

Mode L2 L3 Host * LPM *

0 144K 8k 16k

1 104k 40k 16k

2 72k 72k 16k

3 40k 104k 16k

4 8k 8k 128k

NFE NFE2

NX-OS Routing Modes and Deployment Scenarios

Default Routing Mode: Multi-Chip

• Hierarchical Routing on 95xx/3164 switches.

• Suitable for most deployment scenarios.

• Leverages LC and FM NFE for unicast route programming.

• Provides good balanced scale between Host and LPM prefixes for IPv4 and IPv6.

Nexus 950X/3164 Default Routing Mode: Hierarchical

• V4 & V6 Host Routes • Forwarding Lookup performed in LC NFE. Egress

NFE and Port derived

• Packets are sent directly to Egress NFE (FM NFE

acts as bypass and it performs HiGig switching for

this packet)

• V4 & V6 LPM Routes• No host route match in LC NFE

• LC LPM table has catch all default route to redirect

the packets towards FM NFE for LPM lookup

• FM NFE performs the forwarding lookup to

determine Egress NFE and Port

• Data traffic forwarded to Egress NFE

Line Card

UFT Mode 3

Fabric Module

UFT Mode 4

UFT Mode L3 Host LPM

3 208k 16k *

4 N/A 128k

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 LPM

...

* Catch All Default Route (0/0 and 0::/0) is programmed.

Nexus 950X/3164 : Default Mode - Host Lookup

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

…IPv4: /0 to /31

IPv6: /0 to /127Fabric Modules

IPv4: /32

IPv6: /128

IPv4: 0/0

IPv6: 0/0

Host Route Match ->

Dst NFE, Port

HiGiG

Switching

Packet In

Dest : 1.1.1.1Packet Out

N9K1#show ip route 1.1.1.1

1.1.1.1/32, ubest/mbest: 1/0, attached

*via 10.0.25.4, [20/0], 01:44:03…

N9K1#

Nexus 950X/3164 : Default Mode LPM Lookup

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

…IPv4: /0 to /31

IPv6: /0 to /127Fabric Modules

IPv4: /32

IPv6: /128

IPv4: 0/0

IPv6: 0/0

HOST Lookup: Fail

LPM Lookup: System

Default Route to FM

LPM Lookup

=> Dst NFE,

Port

Packet In

Dest: 2.2.2.2Packet Out

Punt to FM for

LPM lookup



*via 10.0.25.6, [20/0], 01:44:03…

N9K1#

Scale Number : Hierarchical Routing Mode

ASIC V4 Host V6 Host V4 LPM V6 LPM

NFE 208K (48k )̂ 104K 128K/64k 16k/20K +

NFE2 104K (32k )̂ 52K 128K/64k 10k/15K +

N9k-1# show system routing mode

Configured System Routing Mode: Default

Applied System Routing Mode: Hierarchical (Default)

Configured SVI post-routed unknown-unicast hardware flood mode: enabled

N9k-1#

+ “hardware profile ipv6 alpm carve-value <#>” configuration knob

^ ARP/ND

Background and Requirements

• Default Hierarchical Mode is good for most deployments.

• Enhance NX-OS software to further utilize flexibility of Forwarding ASIC.

• Move around UFT Modes and IPv4/IPv6 prefix blocks to address different deployment needs.

• “My Network has around IPv6 60k (<= /64 prefix length), Is there a way to get higher IPv6 LPM scale ?”

• “My deployment is for L2/L3 boundary role and I have large number of hosts behind these Vlans. Is there a way to get higher ARP/ND scale ?”

• “My Network’s routing table size is very small, is there a way to avoid 2 stage look up ?”

Deployment Scenario 1

• Large scale IPv6 LPM (/64) prefixes with some IPv4 LPM as well.

• Below is prefix scale with some growth room built in.

IPv4: Host: <1k LPM: 2k

IPv6: Host: <1k LPM: 60k (< /64) + 2k (/65 to /127)

Nexus 950X/3164 Hierarchical 64B ALPM

• V4 & V6 Host Routes • Forwarding Lookup performed in LC NFE,

Egress NFE and Port derived

• Packets are sent directly to Egress NFE (FM

NFE acts as bypass and it performs HiGig

switching for this packet)Line Card

UFT Mode 3

Fabric Module

UFT Mode 4

64Bit


3 208k 16k

4 N/A 128k

IPv4 Host

IPv6 Host

...

Nexus 950X/3164 Hierarchical 64B ALPM• V4 LPM & V6 <= /64 prefix length LPM Routes

• LC LPM table has default route to punt the

packets towards FM NFE for LPM lookup

• FM NFE performs the forwarding lookup to

determine Egress NFE and Interface


• V6 LPM prefix /65 to /127

• Programmed in LC NFE

• FM NFE can take only V6 prefix length <= /64

• IPv6 prefix length /65 to /127 programmed in

LC NFE in the LPM table. Hence scale limited

to maximum of 3k

• IPv6 prefixes matching LPM prefixes send

directly to Egress NFE


3 208k 16k

4 N/A 128k

Line Card

UFT Mode 3

Fabric Module

UFT Mode 4

64Bit

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 /64 LPM

...IPv6 /65 to /127 LPM

Nexus 950X/3164 : 64B ALPM - Host Lookup

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

… Fabric Modules

IPv4: 0/0

IPv6: 0/0

Host Route Match ->

Dst NFE, Port

HiGiG

Switching

Packet In

Dest: 1:1:1::1Packet Out

IPv4: /32

IPv6: /128

IPv6: /65 to /127

IPv4: /0 to /31

IPv6: /0 to /64

N9K1#show ipv6 route 1:1:1.:1

1:1:1::1/128, ubest/mbest: 1/0, attached

*via 4001::1, [20/0], 01:44:03…

N9K1#

Nexus 950X/3164 : 64B ALPM LPM (V4 <=/31 and V6 <= /64)

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

…IPv4: /0 to /31

IPv6: /0 to /64 Fabric Modules

IPv4: /32

IPv6: /128, /65 to /127

ALPM

Lookup =>

Dst NFE, Port

Packet In


Punt to FM for

LPM lookup

HOST Lookup: Fail

LPM Lookup: System

Default Route to FM

IPv4: 0/0

IPv6: 0/0

N9K1#show ipv6 route 2:2:2::1

2:2:2::/48, ubest/mbest: 1/0, attached

*via 2001::1, [20/0], 01:44:03…

N9K1#

Nexus 950X/3164 : 64B ALPM – V6 LPM (/65 to /127)

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

… Fabric Modules

IPv4: 0/0

IPv6: 0/0

HOST Lookup: Fail

LPM Lookup: Route

Match -> Dst NFE, Port

HiGiG

Switching

Packet InDest: 3:3:3:3:3::3

Packet Out

IPv4: /32

IPv6: /128, /65 to /127

IPv4: /0 to /31

IPv6: /0 to /64

N9K1#show ipv6 route 3:3:3:3:3::3

3:3:3:3:3::/80, ubest/mbest: 1/0, attached

*via 2002::1, [20/0], 01:44:03…

N9K1#

Scale Number : 64B ALPM ModeASIC V4 Host V6

Host

V4 LPM V6 LPM

(<=64)

V6 LPM

(65-127)

NFE 208K (48k )̂ 104K 128K 80K 1K-3K*

NFE2 104K (32k )̂ 52K 128K 80K 1K-3K*

N9k-1(config)# system routing hierarchical max-mode l3 64b-alpm


Note: This requires copy running-config to startup-config before switch reload.

N9k-1#

* “hardware profile ipv6 lpm-entries maximum <#>” configuration knob

^ ARP/ND


• L2/L3 aggregation role with large scale ARP and ND needs.

• Below is prefix scale with some growth room built in

IPv4: Host(ARP): 60k LPM: 3k

IPv6: Host(ND): 40k LPM: 1k

Nexus 950X Max Host Mode• V4 Host & LPM Routes

• On LC NFE only a V4 default route

programmed. Redirects all V4 hosts & LPM

traffic to FM NFE

• FM NFE does best prefix match for both

host and LPM prefixes

• Packets are sent to Egress NFE Line Card

UFT Mode 2

Fabric Module

UFT Mode 3

IPv4 Host

IPv4 LPM

...

Mode MAC L3 Host LPM

2 160k 144k 16k

3 N/A 208k 16k

Nexus 950X Max Host Mode• V6 Host & LPM Routes

• V6 Hosts and LPM prefixes programmed in

LC NFE

• LC NFE performs the forwarding lookup to

determine Egress NFE and Interface.


directly Line Card

UFT Mode 2

Fabric Module

UFT Mode 3

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 LPM

...


2 160k 144k 16k

3 N/A 208k 16k

Modular Multi-Chip : Max Host Mode – V6 Lookup

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

… Fabric Modules

IPv4: 0/0

V6 Route Match on LC

-> Dst NFE and Port

HiGiG

Switching

Packet In


IPv6: /0 to /128

IPv4: /0 to /32

N9K1#show ipv6 route 1:1:1::1

1:1:1::/48, ubest/mbest: 1/0, attached

*via 2001::1, [20/0], 01:44:03…

N9K1#

Modular Multi-Chip : Max Host Mode – V4 Lookup

NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

…IPv4: /0 to /32Fabric Modules

V4 LPM Route ->

System Default Route

Match on LC

Host/LPM

lookup for V4 ->

Dst NFE, Port

Packet In

Dest: 1.1.1.1Packet Out

Punt to FM for

LPM lookup

IPv6: /0 to /128

IPV4: 0/0 to FM



*via 10.0.25.4, [20/0], 01:44:03…

N9K1#

Scale Number : Max Host Mode

ASIC V4 Host V6

Host

V4

LPM

V6 LPM

(<=64)

V6 LPM

(65-127)

NFE 208K (60K )̂ 104K 16K 6K 1K-3K*

N9k-1(config)# system routing max-mode host



N9k-1#


^ ARP


• Network prefixes are hierarchical and aggregated.

• Below is prefix scale with some growth room built in.

IPv4: Host: 16k LPM: 6k

IPv6: Host: 8k LPM: 3k

• If possible, avoid two stage forwarding lookup.

Nexus 950X/3164 Non Hierarchical Routing Mode• All V4/V6 Unicast forwarding decision done

at LC NFE

• V4/V6 Host & LPM Routes• Forwarding Lookup performed in LC NFE,

Egress NFE and Port derived for all L3

Unicast traffic (both Host and LPM routes)

• Packets are sent directly to Egress NFE (FM

NFE acts as bypass and it performs HiGig

switching for this packet)


3 208k 16k

Line Card

UFT Mode 3

Fabric Module

UFT Mode X

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 LPM

...

Nexus 950X/3164 :Non Hierarchical – Any Lookup

NFE NFE NFE

NFE

Fabric 1

NFE NFE

Fabric 6

NFE

Line Card/IO module

… Fabric Modules

IPv4: /0 to /32

IPv6: /0 to /128

HiGiG

Switching

Packet In Packet Out

Host/LPM Route

Match -> Dst NFE,

Port

Scale Number : Non Hierarchical Routing ModeASIC V4 Host V6 Host V4 LPM V6 LPM

(<=64)

V6 LPM

(65-127)

NFE 208K (48k )̂ 104K 12K/4K 6K/2K 1K-3K*

NFE2 104K (32k )̂ 52K 12K/4K 6K/2K 1K-3K*

N9k-1(config)# system routing non-hierarchical



N9k-1#


^ ARP/ND

Modular Multi-Chip Summary

IPv4 Host IPv4 LPMIPv6 Host IPv6 LPM

Fabric Module

Line Card

NFE

NFE

IPv4 Host IPv6 Host

IPv6 LPMIPv4 LPM

Fabric Module

Line Card

NFE

NFE

IPv4 Host

IPv6 Host

IPv4 LPM

Fabric Module

IPv6 LPM

Default

Mode

Non-Hierarchical

ModeMax-host

Mode

NX-OS Software

Line Card

NFE

NFEIPv4 Host

IPv6 Host IPv6 LPM

IPv4 LPM

Single Chip Routing Modes

Nexus 93xx/31xx/32xx Default Routing

UFT Mode 3

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 LPM


3 96k 208k 16k

Scale Number : 93xx/31xx/32xx – Default Mode

N9k-1# show system routing mode

Configured System Routing Mode: Default

Applied System Routing Mode: Max Host (Default)

N9k-1#


ASIC V4 Host V6

Host

V4 LPM V6 LPM

(<=64)

V6 LPM

(65-127)

NFE 208K (48k )̂ 104K 12K/4K 6K/2K 1K-3K*

NFE2 104K (32k )̂ 52K 12K/4K 6K/2K 1K-3K*

^ ARP/ND

Nexus 93xx/31xx/32xx Max L3 Mode


4 32k 16k 128k

UFT Mode 4

IPv4 Host

IPv4 LPM

IPv6 Host

IPv6 LPM

Scale Number: 93xx/31xx/32xx – Max L3

ASIC V4 Host V6 Host V4 LPM V6 LPM

(0-127)

NFE 16K 8K 128K 16K-20K +

NFE2 8K 4K 128K 10K-15K +

N9k-1(config)# system routing max-mode l3



N9k-1#

+ “hardware profile ipv6 alpm carve-value <#>” configuration knob

Walk Through of Commonly used CLIs

System Routing ModeN9K1# show system routing mode Configured System Routing Mode: Default



N9K1# conf terminal Enter configuration commands, one per line. End with CNTL/Z.

N9K1#(config)# system routing non-hierarchical Warning: The command will take effect after next reload.


N9K1#(config)# end

N9K1# show system routing mode Configured System Routing Mode: Non-Hierarchical



N9K1#

Switch is not reloaded after mode change

IPv4 Host Route With Single Nexthop Example



*via 10.0.25.4, [20/0], 01:44:03, bgp-300, external, tag 65012

N9K1#show forwarding ip route 1.1.1.203/32 module 1

IPv4 routes for table default/base

------------------+-----------------------------------------+----------------------+-----------------+-----

Prefix | Next-hop | Interface | Labels | Partial Install

------------------+-----------------------------------------+----------------------+-----------------+------

*1.1.1.203/32 10.0.25.4 Ethernet1/1

/32 Host Route

NextHop Physical Interface

Forwarding Routing Modes VerificationN9K1#show forwarding ip route summary module 1

Module Type : Line-Card

Module Mode : Mode-3

Module Route Download-type : Host only (IPv4+IPv6) (1)

…

N9K1#show forwarding ip route summary module 22

Module Type : Fabric-Module

Module Mode : ALPM (Mode-4)

Module Route Download-type : LPM only (IPv4+IPv6) (2)

…

N9K1#

Line Card is running in Mode 3

Fabric Module is running in Mode 4

bcm-shell command reference

Command Usage

l3 l3table show Display IPv4 Host Entries

l3 defip show Display IPv4 LPM Entries

l3 ip6host show Display IPv6 Host Entries

l3 ip6route show Display IPv6 LPM Entries

l3 multipath show Display ECMP object and Nexthop List

l3 egress show Display Egress object with rewrite information

show c rpkt Display Received packets in Hardware

show c tpkt Display Transmitted packets in Hardware

IPv4 Host Route With Single Nexthop Example

N9K1#bcm-shell module 1 "0:l3 l3table show" | inc 1.1.1.203

211300 1 1.1.1.203 00:00:00:00:00:00 100009 0 0 0 n

N9K1#bcm-shell module 1 "0:l3 egress show 100009"Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC

100009 60:73:5c:fe:e4:6e 4095 4239 13 2 -1 no no no

N9K1#show system internal ethpm info interface ethernet 1/1 | grep -i STATIC

IF_STATIC_INFO: port_name=Ethernet1/1,if_index:0x38078000,ltl=5952,slot=0,

nxos_port=48,dmod=2,dpid=13,unit=1,queue=2160,xbar_unitbmp=0x2,ns_pid=255,slice_num

=0,port_on_slice=0,src_id=0

Unique hardware

port identifier

Egress Object

Nexthop MAC

IPv4 LPM Route With ECMP Example





N9K1#show forwarding ip route 2.2.2.0/24 module 21

IPv4 routes for table default/base

------------------+-----------------------------------------+----------------------+-----------------+-----

Prefix | Next-hop | Interface | Labels | Partial Install

------------------+-----------------------------------------+----------------------+-----------------+------

*2.2.2.0/24 10.0.25.4 Ethernet1/1

10.0.25.6 Ethernet1/2

/24 LPM Route

NextHop Physical Interfaces

IPv4 LPM Route With ECMP Example N9K1#bcm-shell module 21 "0:l3 defip show" | inc 2.2.2.0/24

11012 1 2.2.2.0/24 00:00:00:00:00:00 200256 0 0 0 n

N9K1#bcm-shell module 21 "0:l3 multipath show 200256"

Multipath Egress Object 200256

Interfaces: 100009 100010

N9K1#bcm-shell module 21 "0:l3 egress show 100009"Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC

100009 60:73:5c:fe:e4:6e 4095 4239 13 2 -1 no no no

N9K1#bcm-shell module 21 "0:l3 egress show 100010"

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC

100010 60:73:5c:fe:e4:6e 4095 4240 14 2 -1 no no no

ECMP Object

Egress Objects

Layer 3 Consistency CheckerN9K1# test forwarding inconsistency module 1

Consistency check started.

N9k-1# show forwarding inconsistency module 1

IPV4 Consistency check : table_id(0x1) slot(1)

Execution time : 15118 ms ()

No inconsistent adjacencies.

No inconsistent routes.

Consistency-Checker: PASS for 1

N9k-1# test forwarding ipv6 inconsistency module 1

V6 Consistency check started.

N9k-1# show forwarding ipv6 inconsistency module 1

IPV6 Consistency check : table_id(0x80000001) slot(1)

Execution time : 3130 ms ()

No inconsistent adjacencies.

No inconsistent routes.

Consistency-Checker: PASS for 1

N9k-1#

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference

for full access to session videos and

presentations.

www.CiscoLiveAPAC.com

Give us your feedback and receive a

Cisco 2016 T-Shirt by completing the

Overall Event Survey and 5 Session

Evaluations.– Directly from your mobile device on the Cisco Live

Mobile App

– By visiting the Cisco Live Mobile Site http://showcase.genie-connect.com/ciscolivemelbourne2016/

– Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected Friday 11 March

at Registration

http://www.ciscoliveapac.com/

http://showcase.genie-connect.com/ciscolivemelbourne2016/

Thank you

Layer 3 Forwarding and Troubleshooting...

Documents

Transcript of Layer 3 Forwarding and Troubleshooting...