Troubleshooting Cisco Catalystd2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/BRKCRS-3143.pdf ·...

Troubleshooting Cisco Catalyst 6500 / 6800 Series Switches

Shashank Singh, Customer Support Engineer, Cisco Services

[email protected]

BRKCRS-3143

Agenda

• Architecture: Sup720 vs Sup2T

• Troubleshooting Unicast Forwarding

• Troubleshooting Multicast Forwarding

• Introduction to Instant Access (IA) Solution

• Troubleshooting 6800ia Solution

Goal of this Session …..

Teach commonly used techniques and commands to troubleshoot Cisco Catalyst 6500/6800 switches and …... make it less of a BLACK BOX !!

Why this session covers both Catalyst 6500 and 6800 ?

• Catalyst 6800 is the foundation of Instant Access (IA) solution.

• Catalyst 6800 switch and modules are built on Sup2T/PFC4 architecture

Take Away: Whatever we learn in this session is applicable to Catalyst 6500 Sup2T standalone, VSS and Catalyst 6800 Instant Access Solution.

Catalyst 6500 Series

Catalyst 6800 Series

Architecture: Sup720 vs. Sup2T

Acronyms LegendPFC: Policy Feature Card

DFC: Distributed Forwarding Card

FE: Forwarding Engine

CAM: Content Addressable Memory

TCAM: Ternary or Tertiary CAM

FIB: Forwarding Information Base

ACL: Access Control List

ACE: Access Control Entry

EOBC: Ethernet Out-of-Band Channel

BD: Bridge Domain

LIF: Logical Interface

CoPP: Control Plane Policing

FPOE: Fabric Port of Exit

Reference slide

Supervisor 720/PFC3 Architecture

DBUS

PFC3

SFPSFP /

GETX

Traces # 1 to 16RBUSEOBC

MSFC 3Flash

DRAM

Flash

DRAM MET

QoSTCAM

FIBTCAM

NetFlow

16 Gbps Bus

L3/4 Engine

L2 Engine

AdjTCAM

L2 CAM (64K)

ACECounter

ACLTCAM

20 Gbps

Fabric Interface

and

Replication Engine

Switch Fabric

18 x 20G Traces

1 GbpsSP

CPU

1 GbpsRP

CPU Port ASIC

Layer3 Control-plane

E.g., OSPF, BGP, SNMP

L3/4 forwarding

L2 forwarding

Replication engine

E.g., Multicast, SPAN

Integrated Switch Fabric

Layer2 Control-plane

E.g., LACP, BPDU and

hardware programming

DBUS

PFC4

1GE / 10GE

Uplinks

Traces # 1 to 24RBUSEOBC

MSFC 5

FlashDRAM

CL1TCAM

NetFlow

L3/4 Engine

L2 Engine

L2 CAM (128K)

ACECounter

CL2TCAM

40 Gbps

Fabric Interface

&

Replication EngineSwitch Fabric

26 x 40G Traces

2 GbpsCentral

Management

Processor

CPU

Port ASIC

MET

FIBTCAM

ADJTCAM

LIF Table

LIF Stats

RPF Table

LIF MAP

MSFC5 Complex contains single

dual-core CPU for both Layer 2 and

Layer 3 control-plane protocols and

hardware programming

L3/4 forwarding

L2 forwarding

Replication engine

E.g., Multicast, SPAN

Integrated Switch Fabric

Supervisor 2T/PFC4 Architecture

Supervisor Engine Sup2T

• Sup2T Architecture – White Paper: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11-676346.html

• Sup2T FAQs: http://www.cisco.com/en/US/prod/collateral/modules/ps2797/ps11878/qa_c67-648478.html

• Catalyst 6500 Ethernet Modules Data Sheet: http://www.cisco.com/en/US/products/hw/switches/ps708/products_data_sheets_list.html#anchor3

• Cisco Catalyst 6500 Sup720 to Sup2T engines: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/guide_c07-717261.html

• Cisco Catalyst 6800 Data Sheets and Literature: http://www.cisco.com/c/en/us/products/switches/catalyst-6800-series-switches/literature.html

• Cisco Catalyst 6800 Series Switches – Support Page:ttp://www.cisco.com/c/en/us/support/switches/catalyst-6800-series-switches/tsd-products-support-series-home.html

Reference Materials

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11-676346.html

http://www.cisco.com/en/US/prod/collateral/modules/ps2797/ps11878/qa_c67-648478.html

http://www.cisco.com/en/US/products/hw/switches/ps708/products_data_sheets_list.html

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/guide_c07-717261.html

http://www.cisco.com/c/en/us/products/switches/catalyst-6800-series-switches/literature.html

http://www.cisco.com/c/en/us/support/switches/catalyst-6800-series-switches/tsd-products-support-series-home.html

Troubleshooting Unicast Forwarding


• L2 Topology and Packet Flow

• L2 Packet Flow TroubleshootingL2 CAM, Interface counters/errors, Switch Fabric


• L3 Packet Flow TroubleshootingFIB and Adjacency TCAM

Agenda

L2 Unicast Traffic

• DUT is the Device Under Test we are troubleshooting

• DUT is a 6509-E with Supervisor 2T

• Four TenGigabitEthernet L2 Etherchannel (R1 DUT)

• Four TenGigabitEthernet L2 Etherchannel (DUT R2)

Topology

Po11

Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Po12

Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6

Po12

Host 1

192.168.0.2(0006.5bbc.81a2)

Host 2

192.168.0.3

(0006.5bbc.7acb)

Vlan 10

Both MAC addresses are learned on Port-Channels; Which physical link in the

channel Is actually receiving the packets?

Sup2T# show mac address-table address 0006.5bbc.81a2

Legend: * - primary entry

age - seconds since last seen

n/a - not available

S - secure entry

R - router's gateway mac address entry

D - Duplicate mac address entry

Displaying entries from DFC linecard [1]:

vlan mac address type learn age ports

----+----+---------------+-------+-----+----------+--------

* 10 0006.5bbc.81a2 dynamic Yes 5 Po11



----+----+---------------+-------+-----+----------+--------

* 10 0006.5bbc.81a2 dynamic Yes 90 Po11

L2 Unicast TrafficWhere are the MAC Addresses Learned?

Host 1

Host 2

Sup2T# show mac address-table address 0006.5bbc.7acb



----+----+---------------+-------+-----+----------+--------

* 10 0006.5bbc.7acb dynamic Yes 0 Po12



----+----+---------------+-------+-----+----------+--------


L2 Unicast TrafficWhich Link in the EtherChannel Is Being Used?

Po11Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Po12Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6

Po12

Host 1

192.168.0.2Host 2

192.168.0.3

Gig4/1

R1#show etherchannel load-balance module 4

EtherChannel Load-Balancing Configuration:

src-dst-ip vlan included

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

R1# show etherhannel load-balance interface po11 ip 192.168.0.2 192.168.0.3

Computed RBH: 0x3

Would select Te1/8 of Po11

Mode is “src-dst-ip”. Only use src and dest

IP as argument. Prior to 12.2(33)SXH, use

test etherchannel load-balance …(same

arguments) on the SP, for Sup720 engines.

Link selected is Ten1/8 in Po11 of R1 for traffic to 192.168.0.3

Check load balancing configuration

Use ingress Module number in command

in case per-module load-balancing is

configured (SXH images and later)

L2 Unicast Traffic Network Path Verification: Result

Po11Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Po12Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6

Po12

Host 1

192.168.0.2Host 2

192.168.0.3

Gig4/1

Po11Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Po12Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6

Po12

Host 1

192.168.0.2Host 2

192.168.0.3

Gig4/1

Each direction can use different links in the bundles !






Agenda

Layer 2 Learning and Forwarding• In Sup720 engines, Layer 2 forwarding is based on {VLAN, MAC} pairs. In

Sup2T engines, {port_index, MAC} is used by Bridge Domain(BD) for Bridging and by Logical Interface (LIF) for routing.

• MAC learning is done per PFC or DFC• Each PFC/DFC maintains separate L2 CAM table

• PFC and DFCs age entries independently• Refreshing of entries based on “seeing” traffic from specific host

• New learns on one forwarding engine communicated to other engines via MAC-Sync process (which occurs over EOBC)

Detailed L2 Packet Flow TroubleshootingAre We Learning MAC Addresses?

Sup2T# show mac address-table address 0006.5bbc.7acb vlan 10 [all]

Legend: * - primary entry

age - seconds since last seen

n/a - not available

S - secure entry

R - router's gateway mac address entry

D - Duplicate mac address entry



----+----+---------------+-------+-----+----------+---------




----+----+---------------+-------+-----+----------+----------


Displaying entries from active supervisor:


----+----+---------------+-------+-----+----------+----------

10 0006.5bbc.7acb dynamic Yes 205 Po12

By default, Sup2T prints entries

from all DFCs. In Sup720, use

all keyword to see entry from all

DFCs in the system.

* Denotes the primary

forwarding entry. This entry is owned by ingress forwarding engine for frames sourced from that ethernet address.

Flooding can occur if

MACs are not known

by ALL FEs in the

system

Detailed L2 Packet Flow TroubleshootingVerify L2 Tables: MAC Sync Feature

Sup2T# show mac address-table synchronize statistics

MAC Entry Out-of-band Synchronization Feature Statistics:

---------------------------------------------------------

Module [1]

-----------

Module Status:

Statistics collected from module : 1

Global Status:

Status of feature enabled on the switch : on

Default activity time : 160

Configured current activity time : 160

Statistics from ASIC 0 when last activity timer expired:

Age value in seconds from age byte register : 0x4C

<snip>

Number of entries created new : 377

Number of entries create failed : 0

Module [2]

-----------

Module Status:

Statistics collected from module : 2

Global Status:

Status of feature enabled on the switch : on

Off by default in Sup720. When WS-X6708 is present, it is on by default, and set the mac aging timer to 480 sec. Why 480 ?

Default value is 160 seconds; normal aging timer should be at least 3x activity interval … so with default of 160 sec, change mac aging timer to 480 sec or more

Number of entries that were synced by SW sync feature

Out-of-Band (OOB) MAC-Sync feature is enabled by default in Sup2T. By default, it is disabled in Sup720. Flooding can occur when L2 CAM tables are not in sync. Enable this feature with “mac-address-table synchronize” command (under “config t”) in Sup720.

Ten2/5

DFC4

WS-X6908

Module 2

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

Switch Fabric

WS-X6908

Module 1

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

DFC4

Layer 3/4

Engine

Layer 2

Engine

Layer 2

Engine

Layer 3/4

Engine

Ten1/1

Look at the interface

counters and errors

for the ingress and

egress interfaces

Host1

Host2

Check the L2

forwarding engine

counters

Verify the fabric

channels used in the

flow

Ten1/2

Ten2/6

Detailed L2 Packet Flow Troubleshooting

Detailed L2 Packet Flow TroubleshootingVerify L2 Counters: Interface Counters

Sup2T# show interface ten 1/2 counters

Port InOctets InUcastPkts InMcastPkts InBcastPkts

Te1/2 249784 2000 8 40

Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts

Te1/2 83246 18 6 0

Sup2T#show interface ten 1/1 counters


Te1/1 10590 18 28 0


Te1/1 246412 2008 10 0

Sup2T#show interface ten 2/5 counters


Te2/5 2890 2890 0 0


Te2/5 273441 2032 11 0

And, similarly on Ten2/5

Did a ping (2000 packets/100 bytes per packet) from 192.168.0.2 to 192.168.0.3.

verify interface counters relevant to the path did move sufficiently !!

shows

interface level

packet counts

and errors

since last time

clear counters

was issued.

Hardware counters. Not cleared by

“clear counters” command.

Detailed L2 Packet Flow TroubleshootingVerify L2 Counters: Interface Counters

Sup2T# show interface ten1/1 counter error

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

Te1/1 0 0 0 0 0 0

Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants

Te1/1 0 0 0 0 0 0 0

Port SQETest-Err Deferred-Tx IntMacTx-Err IntMacRx-Err Symbol-Err

Te1/1 0 0 0 0 0

Sup2T# clear counters

Sup2T# show counter interface te1/1 delta

Time since last clear

---------------------

00:00:02

64 bit counters:

0. rxHCTotalPkts = 1

1. txHCTotalPkts = 3

2. rxHCUnicastPkts = 0

3. txHCUnicastPkts = 0

<snip>

Sup2T# show counter interface te1/1

<snip>

64 bit counters:

0. rxHCTotalPkts = 13021673

1. txHCTotalPkts = 3090200

2. rxHCUnicastPkts = 2684645

3. txHCUnicastPkts = 2684649

<snip>

.... nearly 140 counters ...

shows traffic statistics since last clear

Detailed L2 Packet Flow TroubleshootingVerify L2 Counters: L2 Forwarding Engine VLAN Count

Sup2T #show vlan id 10 counters

* L2 counters include multicast and broadcast packets

Vlan Id : 10

L2 Unicast Packets : 4012

L2 Unicast Octets : 401868

L3 Input Unicast Packets : 0

L3 Input Unicast Octets : 0

L3 Output Unicast Packets : 0

L3 Output Unicast Octets : 0

L3 Output Multicast Packets : 0

L3 Output Multicast Octets : 0

L3 Input Multicast Packets : 0

L3 Input Multicast Octets : 0

L2 Multicast Packets : 0

L2 Multicast Octets : 0

VLAN is bidirectional, so counts

both directions of the flow

(192.168.0.2 192.168.0.3)

Sup2T# show fabric fpoe map

slot channel logical fpoe physical fpoe

1 0 0 5

1 1 1 5

1 2 32 6

1 3 33 6

2 0 2 11

2 1 3 11

2 2 34 7

2 3 35 7

<snip>

For each ingress and

egress interface: find

mapping between

interface and Fabric

Port Of Exit (FPOE).

Detailed L2 Packet Flow TroubleshootingIdentifying the Fabric Channels

Sup2T# sh fabric fpoe interface ten1/1

fpoe for TenGigabitEthernet1/1 is 1






fpoe for TenGigabitEthernet2/6 is 34Find mapping between

FPOE and Slot/Channel

(requires service internal

config under config t)

Ten2/5

DFC4

WS-X6908

Module 2

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

Switch Fabric

WS-X6908

Module 1

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

DFC4

Layer 3/4

Engine

Layer 2

Engine

Layer 2

Engine

Layer 3/4

Engine

Ten1/1

Host1

Host2

Ten1/2

Ten2/6

FPOE 1

FPOE 34

Fabric ASIC

Fabric Interface ASIC

Detailed L2 Packet Flow TroubleshootingVerify L2 Counters: Switching Fabric Utilization

Sup2T# show fabric utilization detail

Fabric utilization: Ingress Egress

Module Chanl Speed rate peak rate peak

1 0 40G 0% 0% 0% 0%

1 1 40G 0% 1% @15:47 21Feb12 0% 0%

2 0 40G 0% 0% 0% 0%

2 1 40G 0% 0% 0% 1% @02:34

22Feb12

5 0 20G 0% 0% 0% 0%

5 1 20G 0% 0% 0% 0%

<snip>

Sup2T# show fabric status

slot channel speed module fabric hotStandby Standby Standby

status status support module fabric

1 0 40G OK OK Y(not-hot)




5 0 20G OK OK N/A

5 1 20G OK OK N/A

6 0 20G OK OK N/A

6 1 20G OK OK N/A

Check utilization (current and last peak

value) for relevant fabric channels … did

any peak coincide with moment of drops?

Check status of fabric channels is

OK. An example for misbehaving

module or fabric channel: Module

status is reported as “DDR Sync”

Detailed L2 Packet Flow TroubleshootingVerify L2 Counters: Relevant Fabric Channel Counters

Sup2T# show fabric channel-counters 1

slot channel rxErrors txErrors txDrops lbusDrops

1 0 0 0 0 0

1 1 0 0 0 0

Sup2T# show fabric channel-counters 2

slot channel rxErrors txErrors txDrops lbusDrops

2 0 0 0 0 0

2 1 0 0 0 0

Sup2T# show fabric errors 1

Module errors:

slot channel crc hbeat sync DDR sync

1 0 0 0 0 0

1 1 0 0 0 0

Fabric errors:

slot channel sync buffer timeout

1 0 0 0 0

1 1 0 0 0

fabric ASIC unable to send traffic to the fabric enabled module for last +3 seconds

fabric serial link bit errors (8 serial links in

each fabric channel), reported as soon as 2

fabric serial link interrupts within 100ms; can

result in rxErrors / txErrors; check card

inserted OK ?

line card fabric ASIC reports bad

packets: card inserted properly ? A

few incrementing ‘rxErrors', which is

not correlated to any network events,

is OK & acceptable.

unable to send packets from fabric to line

card: Check traffic levels, line card OK ?

fabric interface unable to send

packets from local bus to fabric

(Supervisor and 65XX modules only,

67XX and above report Overruns in

“show interface” results. check traffic

levels, signs of congestion ?






Agenda

L3 Unicast Traffic Network Configuration

• DUT is the Device Under Test we are troubleshooting

• DUT is a 6509-E with Supervisor 2T

• Four TenGigabitEthernet L2 Etherchannel Trunk (R1 DUT)

Vlan 10, 20, 30 and 40 are assigned with 192.168.10.0/24, 192.168.20.0/24, 192.168.30.0/24 and 192.168.40.0/24 subnets respectively.

• Four L3 Links (DUT R2)

Four links are assigned with 172.16.10.0/24, 172.16.20.0/24, 172.16.30.0/24 and 172.16.40.0/24 subnets respectively.

Po11Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6

VLANS 10,20,30 and 40

Host2

100.100.100.1

Host1

L3 Links200.200.200.1

L3 Unicast TrafficDifferent Switching Paths for L3 Traffic in Catalyst 6500/6800

Process Switching Path

Software-based CEF Switching Path

Hardware-based CEF switching Path

DUT

This slide is just a logical representation of

different switching paths (also known as

Switching Vectors) in Catalyst 6500/6800.

Host1 Host2

L3 Unicast TrafficHost 1 Host 2: Which L3 Next Hop / L2 Link from R1?

SWR1# show ip route 200.200.200.1

Routing entry for 200.200.200.1/32

Known via "ospf 100", distance 110, metric 3, type intra area

Last update from 192.168.40.1 on Vlan40, 00:10:12 ago

Routing Descriptor Blocks:

192.168.40.1, from 192.168.0.2, 00:10:12 ago, via Vlan40

Route metric is 3, traffic share count is 1

192.168.30.1, from 192.168.0.2, 00:10:12 ago, via Vlan30


* 192.168.20.1, from 192.168.0.2, 00:10:12 ago, via Vlan20


192.168.10.1, from 192.168.0.2, 00:10:12 ago, via Vlan10


R1# show ip cef exact-route 100.100.100.1 200.200.200.1

100.100.100.1 -> 200.200.200.1 => IP adj out of Vlan40, addr 192.168.40.1

R1# show mls cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Vl10, Next Hop: 192.168.20.1, Vlan: 10, Destination Mac: b414.8961.3780

R1-sp# test etherchannel load-bal int port-ch 11 ip 100.100.100.1 200.200.200.1

Computed RBH: 0x7

Would select Te1/8 of Po11

Next hop used for HW based CEF

(HW forwarding path). Note: “0” is

used for both src and dest L4 port

numbers as test flow was ICMP echo

Command is available only in SP console on

sup720 (remote login switch). Check which link

between R1 and DUT is chosen.

HW

Equal Cost Routes to the

destination prefix

Next hop used for SW based

CEF (SW forwarding data path)

Note: R1 is a

Cat6500 with

Sup720, which

supports “mls”

commands.

* denotes the path it takes for the next process-

switched traffic. It moves in a round-robin fashion,

L3 Unicast Traffic Host 1 Host 2: Which L3 Next Hop from DUT?

Sup2T# show ip route 200.200.200.1



Last update from 172.16.20.2 on TenGigabitEthernet1/6, 00:36:01 ago


172.16.40.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet2/6






* 172.16.10.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet1/5


Sup2T# show ip cef exact-route 100.100.100.1 200.200.200.1

100.100.100.1 -> 200.200.200.1 => IP adj out of TenGigabitEthernet1/6, addr 172.16.20.2

Sup2T# show plat hardware cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Te2/6, Next Hop: 172.16.40.2, ifnum: 0x12, Destination Mac: f866.f2d2.fa80

LIF: 0x20004013

Next hop used for SW based

CEF (SW forwarding data path)

Equal Cost Routes to

the destination prefix

Next hop used for HW based CEF (HW forwarding

path). Note: “0” is used for both src and dest L4

port numbers as test flow was ICMP echo

SW

HW

L3 Unicast Traffic Network Path Verification: Result

Po11

Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6Host 1

100.100.100.1Host 2

200.200.200.1

Po11

Ten1/4 Ten1/1

Ten1/2Ten1/8

Ten2/1Ten1/5

Ten2/2Ten1/7

Po11

R1 DUT R2

Ten1/5 Ten1/3

Ten1/4Ten1/6

Ten1/7Ten2/5

Ten1/8Ten2/6Host 1

100.100.100.1Host 2

200.200.200.1

Each direction can use different links in the bundles !

What Did We Get from Path Verification?

• The physical links the specific traffic flow should come in and leave the DUT.

• Helps us to isolate if there is any faulty or oversubscribed interface.

• Caveats:

• Flapping links in port channel, can change the bundle hash mapping, and change physical path of traffic

• Clearing routes can as well change the order in which the L3 adjacencies get re-programmed, and in case of ECMP hence change the physical path of the traffic

• Any of these happen, you need to re-verify the path






Agenda

Ten2/5

DFC4

WS-X6908

Module 2

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

Switch Fabric

WS-X6908

Module 1

Port

ASIC

Fabric

Interface &

Replication

Engine

MET

Port

ASIC

DFC4

Layer 3/4

Engine

Layer 2

Engine

Layer 2

Engine

Layer 3/4

Engine

Ten1/1

Host1

Host2

Check the L3 / L4

forwarding engine

Ten1/2

Ten2/6

Detailed L3 Packet Flow TroubleshootingL3 FIB Table Programming Flow


• L3 forwarding tables get programmed by SW: copy of SW forwarding tables in HW

• EOBC is used for communication between modules and RP, and program L3 tables

L3/4 Engine in Detail: Counters and Tables

PFC4

CL1TCAM

NetFlow

L3/4 Engine

L2 Engine

L2 CAM (128K)

ACECounter

CL2TCAM

FIBTCAM

ADJTCAM

LIF Table

LIF Stats

RPF Table

LIF MAP

DBUS

RBUSEOBC

remote command module <mod> show ip cef

remote command module <mod> show adjacency detail

show ip arp

show ip cef adjacency

FIB / Adjacency TablesL3 FIB Table Programming Flow in Sup2T

show ip route (RIB) IOS® Routing Table (RP)

IOS FIB Table (RP)

IOS FIB Table (PFC/DFC)

FIB Table (PFC/DFC)

Verify Layer 3 rewrite Verify Layer 2 rewrite

IOS ARP Cache Table (RP)

IOS Adjacency Table (RP)

IOS Adjacency Table (PFC/DFC)

Adjacency Table (PFC/DFC)

show ip cef

show plat hard cef lookup <ip address> <mod>

show plat hard cef adjacency entry





Last update from 192.168.40.2 on Vlan40, 00:00:19 ago


192.168.40.2, from 192.168.252.10, 00:00:19 ago, via Vlan40


192.168.30.2, from 192.168.252.10, 00:00:19 ago, via Vlan30


192.168.20.2, from 192.168.252.10, 00:00:19 ago, via Vlan20


* 192.168.10.2, from 192.168.252.10, 00:00:29 ago, via Vlan10


Verify IP Routing Table

Host 2

Host 1




Last update from 172.16.30.2 on TenGigabitEthernet2/5, 00:01:00 ago


* 172.16.40.2, from 192.168.0.2, 00:01:10 ago, via TenGigabitEthernet2/6








SW

SW

Detailed L3 Packet Flow TroubleshootingL3 FIB Table and Counters

SWSup2T# show ip cef 200.200.200.1

200.200.200.1/32

nexthop 172.16.10.2 TenGigabitEthernet1/5




Sup2T# show ip cef exact-route 100.100.100.1 src-port 0 200.200.200.1 dest-port 0

100.100.100.1 -> 200.200.200.1 => IP adj out of TenGigabitEthernet1/6, addr 172.16.20.2

Sup2T# show ip cef adjacency tengig 1/6 172.16.20.2

172.16.20.2/32

attached to TenGigabitEthernet1/6

200.200.200.1/32


IP CEF entries for destination IP addr

IP CEF Adjacency entries for next-hop IP address

IP CEF entries for destination IP address

Sup2T# show platform hardware cef lookup 200.200.200.1

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

8080 200.200.200.1/32 Te1/5 ,f866.f2d2.fa80 (Hash: 0001)

Te1/6 ,f866.f2d2.fa80 (Hash: 0002)

Te2/5 ,f866.f2d2.fa80 (Hash: 0004)

Te2/6 ,f866.f2d2.fa80 (Hash: 0008)

Sup2T# show platform hardware cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Te2/6, Next Hop: 172.16.40.2, ifnum: 0x12, Destination Mac:

f866.f2d2.fa80 LIF: 0x20004013


HWNo more MLS for Sup2T engines. For Sup720, use “mls” instead of “platform hardware”.


Sup2T# show adjacency TenGigabitEthernet1/6 172.16.20.2 detail

Protocol Interface Address

IP TenGigabitEthernet1/6 172.16.20.2(14)

0 packets, 0 bytes

epoch 0

sourced in sev-epoch 0

Encap length 14

F866F2D2FA80B414896137800800

L2 destination address byte offset 0

L2 destination address byte length 6

Link-type after encap: ip

ARP

SW

HWSup2T# show platform hardware cef ip 200.200.200.1 detail module 1

Codes: M - mask entry, V - value entry, A - adjacency index, NR- no_route bit

LS - load sharing count, RI - router_ip bit, DF: default bit

CP - copy_to_cpu bit, AS: dest_AS_number, DGTv - dgt_valid bit

DGT: dgt/others value

Format:IPV4 (valid class vpn prefix)

M(8080 ): 1 F 3FFF 255.255.255.255

V(8080 ): 1 0 0 200.200.200.1

(A:311296, LS:3, NR:0, RI:0, DF:0 CP:0 DGTv:1, DGT

Rewrite information (Dmac|Smac|0800): verify it is conform with next hop rewrite info

Ingress module, for the specific flow

Start adjacency pointer is 311296


Sup2T# show platform hardware cef adjacency entr311296 detail module 1

Index: 311296 -- Valid entry (valid = 1) –

Adjacency fields:

___________________________________________________

|adj_stats = EN | fwd_stats = EN | trig = 0

|_________________|__________________|______________

|l3_enable = ON (classify as Layer3) | age = 3

|_________________|__________________|______________

|format = IP | rdt = ON | ignr_emut = 0

|_________________|__________________|______________

|vpn = 0x3FFF | elif = 0x400C | ri = 3

|_________________|__________________|______________

|top_sel = 0 | zone_enf = OFF | fltr_en = OFF

|_________________|__________________|______________

|frr_te = OFF | idx_sel = 0 | tnl_encap = 0

|_________________|__________________|______________

|rw_hint = 0 | ttl_control = 4 |

|_________________|__________________|______________

Format of the packet sent out on

the wire ...

HW

Checking the entry in the ingress module


Rewrite MAC info

RIT fields: The entry has a Layer2 Format

_________________________________________________________

|decr_ttl = YES | pipe_ttl = 0 | utos = 0

|_________________|__________________|____________________

|l2_fwd = 0 | rmac = 0 | ccc = L3_REWRITE

|_________________|__________________|____________________

|rm_null_lbl = YES| rm_last_lbl = YES| pv = 0

|_________________|__________________|____________________

|add_shim_hdr= NO | rec_findex = N/A | rec_shim_op = N/A

|_________________|__________________|____________________

|rec_dti_type = N/A | rec_data = N/A

|____________________________________|____________________

|modify_smac = YES| modify_dmac = YES| egress_mcast = NO

|____________________________________|____________________

|ip_to_mac = NO

|_________________________________________________________

|dest_mac = f866.f2d2.fa80 | src_mac = b414.8961.3780

|___________________________|_____________________________

|

Statistics: Packets = 0

Bytes = 0

Output Continued ….

Increases in the ingress DFC/PFC. Counters will be cleared when adjacency is read.

HW


• Default entry: 0.0.0.0/0 (“match all”)

• Always at bottom of FIB TCAM, if no default route, punt to drop adjacency.

• Drop adjacency (route to Null0): subject to rate limiter "ICMP UNREAC. NO-ROUTE"

L3 FIB Table Special Entries/Adjacencies

Sup2T# show ip route 0.0.0.0 0.0.0.0

% Network not in table

Sup2T# show plat hard cef lookup 123.0.1.1



134368 0.0.0.0/0 drop




134368 0.0.0.0/0 Vl1200 ,0011.bc75.9c00

No default route present

Match-all entry links to drop adjacency, which is subject to rate limiter "ICMP UNREAC. NO-ROUTE“.

In-profile packets get punted to CPU … so possible reason for packets hitting CPU

After adding default route to Vlan 1200, adjacency points to next hop, all switched in HW


• FIB receive (local IP address): subject to rate limiter “CEF RECEIVE”

• Traffic matching CEF GLEAN entry is subject to rate-limiter “CEF GLEAN”

L3 FIB Table Special Entries/Adjacencies



7701 172.16.10.1/32 receive

If not present, packets for local IP addresses don’t get to RP (SW)



Known via "connected", distance 0, metric 0

(connected, via interface)


* directly connected, via TenGigabitEthernet2/6


Sup2T# show ip arp 172.16.40.2

Sup2T#

Sup2T# show plat hard cef look 172.16.40.2



2128 172.16.40.0/24 glean

Known Subnet

If not present, packets to unresolved IP addresses for directly connected hosts/routers will not get punted to CPU (SW) to trigger ARP resolution

Unresolved ARP for directly connected host

Troubleshooting Unicast ForwardingSummary

Determine path-of-the-packet through a

L2 and L3 network

L2 Forwarding

‒ Check MAC Learning

‒ L2 MAC tables are in sync (flooding)

‒ Interface Errors and Statistics

‒ Switch fabric path

L3 Forwarding

‒ SW and HW FIB entries

‒ Adjacency / Rewrite info

It is very critical to determine the flow

experiencing packet loss and find path-

of-the-packet through the network.

Knowledge of switch hardware and

software architecture expedites the

troubleshooting, and helps for timely

resolution of the problem.

L2 and L3 forwarding troubleshooting

for Catalyst 6800 is same as for Sup2T-

based Catalyst 6500.

Take Away Points

Sup2T Unicast Forwarding

• IP Unicast Layer 3 Switching: http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-1SY/config_guide/sup2T/15_1_sy_swcg_2T/cef.html

• Catalyst 6500 Switches ARP and CAM Table Issues Troubleshooting:http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/71079-arp-cam-tableissues.html

• Catalyst 6500 Series Switches – Troubleshooting TechNotes: http://www.cisco.com/c/en/us/support/switches/catalyst-6500-series-switches/products-tech-notes-list.html

• Cisco Catalyst Instant Access – Q&A: http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6800ia-switch/qa_c67-728684.html

Reference Materials

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-1SY/config_guide/sup2T/15_1_sy_swcg_2T/cef.html

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/71079-arp-cam-tableissues.html

http://www.cisco.com/c/en/us/support/switches/catalyst-6500-series-switches/products-tech-notes-list.html

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6800ia-switch/qa_c67-728684.html

Troubleshooting Multicast Forwarding

Multicast Troubleshooting

• Terminology

• Multicast Replication and Modes

• Multicast Forwarding Troubleshooting

Agenda

Terminology

• OIF: Outgoing Interface

• OIL: Outgoing Interface List

• IGMP: Internet Group Management Protocol

• Multicast FIB: Contains the (*,G) and (S,G) entries as well as RPF-VLAN

• Adjacency Table: Contains the rewrite information and MET index

• LTL: Local Target Logic - Forwarding logic for the Catalyst® 6500 / 6800

• MET: Multicast Expansion Table - Hardware table that contains the OIFs for the (*,G) and (S,G) entries

Local Target Logic (LTL)

• Every valid packet that ingresses the Catalyst 6500/6800 will be sent to a forwarding engine (FE) within the system (DFC or the PFC on the supervisor)

• The FE makes the decision about where to forward the packet or to drop the packet

• Part of the result of the forwarding decision is a destination LTL index (or destination index)

• The destination index is used to select the physical port(s) that will forward the packet

• For multicast, another important part of the forwarding decision is the MET index

Multicast Expansion Table (MET)

• MET is memory where the list of OIFs for the multicast entries are stored

• MET block contains the list of OIFs and the corresponding destination LTL index for each

• Each replication engine has a separate MET

• MET index from the CEF adjacency can be used to read the table

• MET tables are independent of the DFC. In other words, even CFC modules have MET tables

maps to port or set of ports


• Terminology



Agenda

Multicast Replication

• Replication: Process of creating copies of packets

• L2 Replication: Creating copies of a packet within a single VLAN (e.g., Forwarding a single broadcast packet out all ports within a VLAN)• Does not require a replication engine

• L3 Replication: Creating copies of a multicast packet for forwarding out each of the interfaces in an OIL• Requires a replication engine

• For this multicast discussion, the term Replication will mean L3 Replication

Ingress Replication Mode

• Replication engine on ingress module performs replication for all OIFs

• One copy of the original packet is forwarded across the fabric for each of the OIFs

• Input and replicated packets get lookup on PFC or ingress DFC

• Default to ingress mode when at least one module not capable of egress mode is present in the system

• MET’s on all replication engines are symmetric or synchronized

Switch

Fabric

Three Packets

Cross Fabric

RE

RE = Replication Engine

RE

RE

RE

1

2

3

4

Egress Replication Mode

• Input packets get lookup on ingress DFC, replicated packets get lookup on egress DFC

• For OIFs on ingress module, local engine performs the replication

• For OIFs on other modules, ingress engine replicates a single copy of packet over fabric to all egress modules

• Engine on egress module performs replication for local OIFs

• MET tables on different modules can be asymmetric

Switch

Fabric

One Packet

Crosses Fabric

RE = Replication Engine

RE

RE

RE

RE

1

2

3

4


• Terminology



Agenda

Diagram for Troubleshooting Example

Gi4/1

VLAN 30

Gi1/1

VLAN 10

Receiver

10.10.30.3

Source: 172.16.10.1

Group: 225.1.1.1

Receiver

10.10.20.3

Gi1/2

VLAN 20Gi4/2

L3 Link Receiver

10.10.40.3

• DUT is a Catalyst 6500 with a Sup2T engine

• Module 1 and 4 are WS-X6824-SFP with DFC4-A.

• Server sending 225.1.1.1 stream, received on Gig1/1 in Vlan 10

• Receivers are connected to module 1 and 4, and in vlan 20, 30 and across an L3 link

DUT

Layer 3 Network

Router

Multicast Replication Modes• In classic system (all modules are non-DFC), replication always occurs on the active supervisor engine

• In a fully fabric-enabled system, there are two possible replication modes: Ingress replication mode

Egress replication mode

Sup2T#show platform hardware capacity multicast

L3 Multicast Resources

Replication mode: egress

Bi-directional PIM Designated Forwarder Table Capacity: 8 Per Vrf

Bi-directional PIM Designated Forwarder Table usage:

Vrf IPV4 used IPV6 used Total used

Replication capability: Module Capability

1 egress

4 egress

6 egress

MET table Entries: Module Total Used %Used

1 65518 4 1%

4 65518 6 1%

6 32752 2 1%

Multicast LTL Resources

Usage: 38848 Total, 581 Used

Capabilities of each module in the system. One

card in the chassis only capable of ingress mode

cause the mode to move to ingress

Use show mls ip multicast

capability in older versions

Shows that the mode

for the system is

Egress

“Router” port indicates that

the CPU is an mrouter port

IGMP Snooping

Use show ip igmp groups [group] to verify that the receivers’ membership reports are

received by the switch

Sup2T#sh ip igmp groups 225.1.1.1

IGMP Connected Group Membership

Group Address Interface Uptime Expires Last Reporter Group Accounted

225.1.1.1 Vlan30 01:44:42 00:02:24 10.10.30.5

225.1.1.1 Vlan20 01:44:42 00:02:17 10.10.20.5

225.1.1.1 GigabitEthernet4/2 01:48:46 00:02:13 10.10.40.3

Membership Reports and L2 Forwarding Table

Use show mac-address-table multicast igmp-snooping to display the

IGMP Snooping L2 forwarding table

Sup2T#sh mac address-table multicast igmp-snooping

vlan mac/ip address LTL ports

+----+-----------------------------------------+------+--------------

20 ( *,225.1.1.1) 0x912 Router Gi1/2

30 ( *,225.1.1.1) 0x914 Router Gi4/1

10 IPv4 OMF 0x90C Router



Gig1/2 and Gig4/1 are receivers

in vlan 20 and 30 respectively

shows ONLY

the last reporter

shows the

receivers in the

VLANs and L3

Interfaces

If a specific vlan is not

listed, then there is an

issue with IGMP SW

SW

RPF neighbor

RPF VLAN

Multicast Forwarding

Sup2T#show ip mroute 225.1.1.1

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

<snip>

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.1.1.1), 02:02:40/stopped, RP 192.168.100.1, flags: SJC

Incoming interface: Null, RPF nbr 10.10.10.5

Outgoing interface list:

Vlan30, Forward/Sparse, 01:50:46/00:02:14


GigabitEthernet4/2, Forward/Sparse, 01:54:50/00:02:11

(172.16.10.1, 225.1.1.1), 01:32:44/00:02:09, flags: JT

Incoming interface: Vlan10, RPF nbr 10.10.10.5

Outgoing interface list:

GigabitEthernet4/2, Forward/Sparse, 01:32:44/00:02:11



(S,G) Entry in SW

(S,G)

OIL

SW

Make sure that drops are not

incrementing. If RPF drops are

seen, do show ip rpf <src-ip-

addr> to verify the RPF

information. Also, do show ip

route <src-ip-addr> to verify the

RPF interface for that multicast

stream

Multicast Forwarding

Sup2T#show ip mroute 225.1.1.1 count

<snip>

Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second

Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 225.1.1.1, Source count: 1, Packets forwarded: 720, Packets received: 720

RP-tree: Forwarding: 3/0/100/0, Other: 3/0/0

Source: 172.16.10.1/32, Forwarding: 717/0/100/0, Other: 717/0/0

Sup2T#show ip mfib 225.1.1.1 count

Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second

Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

<snip>

Group: 225.1.1.1

RP-tree,

SW Forwarding: 0/0/0/0, Other: 0/0/0

HW Forwarding: 3/0/100/0, Other: 0/0/0

Source: 172.16.10.1,

SW Forwarding: 0/0/0/0, Other: 0/0/0

HW Forwarding: 878/0/100/0, Other: 0/0/0

Totals - Source count: 1, Packet count: 881

Forwarded Multicast Packets

Make sure that forwarding packet counts are

incrementing (updated every 10 seconds)

This command is recommended for faster

response, in large-scale deployments.

Packets forwarded in

hardware vs. software

SW

Multicast Forwarding Entry Egress Mode

• The primary entry is used by the ingress forwarding engine for:

• Forwarding to all receivers and mrouters in the ingress VLAN

• Forwarding to all “local” receivers and mrouters on all OIFs in the OIL

• Forwarding a copy of the packet across the switching fabric to egress module(s)

• The secondary (or non-primary) entry is used by the egress forwarding engines for:

• Forwarding to all “local” receivers and mrouters on all OIFs in the OIL

Primary and Secondary Entries


Sup2T-dfc1#sh platform hardware multicast routing ip group 225.1.1.1 detail

IPv4 Multicast CEF Entries for VPN#0

<snip>

(172.16.10.1, 225.1.1.1/32)

FIBAddr: 0x40 IOSVPN: 0 RpfType: SglRpfChk SrcRpf: Vl10

CPx: 0 s_star_pri: 1 non-rpf drop: 0

PIAdjPtr: 0x38001 Format: IP rdt: off elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: on mtu_idx: 2(1518)

PV: 1 rwtype: MCAST_L3_RWT_L2_EXPS

met3: 0x34 met2: 0x28

Packets: 1393 Bytes: 139300

NPIAdjPtr: 0x38002 Format: IP rdt: on elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off

PV: 0 rwtype: MCAST_L3_REWRITE

met3: 0x34 met2: 0x0 DestNdx: 0x7FF3

Packets: 0 Bytes:

Closer Look at the Primary Entry

(S,G) RPF VLAN

Primary Entry (PI)

Access the DFC using “remote

login module X” command.

met3: MET index used to retrieve the LTL

indices for receivers and mrouters local to

the ingress replication engine.


index used to forward a single copy of the

multicast packet across the switching fabric.

Non-primary Entry (NPI)

Number of packets/bytes

forwarded using this entry.

HW


Continued ……




met3: 0x34 met2: 0x0 DestNdx: 0x7FF3

Packets: 0 Bytes:

MET offset: 0x34

OIF AdjPtr Elif CR

+------+----------+--------+---+

Vl20 0x8014 0x14 1T1

MET offset: 0x28

OIF AdjPtr Elif CR

+---------+-------+---------+----+

EDT-34001 0x34001 0x8400A 1T1

Closer Look at the Primary Entry (continued)

met3 Index (from primary entry)

met2 Index (from primary entry)

Vlan 20 (receiver connected to Gig1/2)

For copy of the packet sent

via the switching fabric

HW


Sup2T-dfc4#sh platform hardware multicast routing ip group 225.1.1.1 detail

IPv4 Multicast CEF Entries for VPN#0

<snip>

(10.10.10.5, 225.1.1.1/32)

FIBAddr: 0x0A IOSVPN: 0 RpfType: SglRpfChk SrcRpf: Vl10

CPx: 0 s_star_pri: 1 non-rpf drop: 0

PIAdjPtr: 0x54000 Format: IP rdt: off elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: on mtu_idx: 2(1518)

PV: 1 rwtype: MCAST_L3_RWT_L2_EXPS

met3: 0xA met2: 0x8

Packets: 0 Bytes: 0




met3: 0xA met2: 0x0 DestNdx: 0x7FF3


Closer Look at the Secondary Entry

(S,G)RPF VLAN

Primary Entry (PI)

Access the DFC using “remote

login module X” command.


indices for receivers and mrouters local to

the egress replication engine.


index used to forward a single copy of the

multicast packet across the switching fabric.

Here, met2 = 0, because egress module will

NOT send anything back to fabric for this

specific (S,G) flow.

Non-primary Entry (NPI)

Number of packets/bytes

forwarded using this entry.

HW


Continued ……




met3: 0xA met2: 0x0 DestNdx: 0x7FF3


MET offset: 0xA

OIF AdjPtr Elif CR

+-------------+----------+-----------+------------+

Gig4/2 0x800C 0x408F 4/T1

Vl30 0x801E 0x1E 4/T1

MET offset: 0x8

OIF AdjPtr Elif CR

+-------------+----------+-----------+------------+

EDT-34005 0x5C000 0x840A 4/T

Found 2 entries.

Closer Look at the Secondary Entry (continued)

met3 Index

Receiver connected to Gig4/2

across an L3 interface, and a

receiver on Gig4/1 in vlan 30

HW

Introduction to Instant Access(IA)

Solution

Catalyst Instant Access(IA) Solution

• Introduction and Evolution

• Instant Access (FEX) Discovery

• Instant Access (FEX) Verification

• Forwarding on Instant Access Solution

Agenda

Catalyst Instant Access Evolution

STANDALONE

ACCESS

SWITCH

LACP /

PAGP

ACCESS

SWITCH

VSS

SiSi SiSi

ACCESS

SWITCH ACCESS

SWITCH

LACP /

PAGP

VSL

INSTANT ACCESS

INSTANT

ACCESS

CLIENT

INSTANT

ACCESS

CLIENT

VSL

SiSi SiSi

Traditional Campus with stacking at access

SiSi SiSi

SiSi SiSi

Building 1 Building 2 Building 3 Building 4

Core

SiSi SiSi SiSi SiSi SiSi SiSi

34 Total Devices Management (image and configuration)

48 Access Trunks/Port-Channels

4032 User Ports

VSS Campus with Stacking

Building 1 Building 2 Building 3 Building 4

Core 29 Total Devices for Image and Configuration Management

48 Access Trunks/Port-Channels

4032 User Ports

Catalyst Instant Access

Core

5 Total Devices for Image and Configuration Management

Automated Trunk Configuration

4032 User Ports

Catalyst Instant Access

Core

NO Routing Protocols or Spanning-Tree configuration

between Access and Distribution

NO Trunks to Configure from Access to Distribution

NO Configuration or Image Management at Access

Troubleshooting 6800IA Solution

Terminologies

• FEX – Fabric Extender

• IA Parent – Instant Access Parent / Controller Switch

• IA Client – Instant Access Client / Cat6k Remote Line Card

• SDP – Switch Discovery Protocol

• SRP – Switch Role Protocol

• SCP – Switch Configuration Protocol

• RSL – Remote Satellite Link (fabric link interconnecting IA Parent with IA Client)

• VIF – Virtual Interface (logical representation of FEX physical ports)

• RPF – Route Path Forwarding

• VNTAG – Virtual Native Tagging

Troubleshooting 6800IA solution Instant Access Components

Supervisor 2T

X6904 -16x10G

Supervisor 2T

X6904 -16x10G

Catalyst 6880-X

Catalyst 6807-XLCatalyst 6500E

FEX-Fabric IA Client

(VSS)

(VSS)

(VSS)

IA Parent

Instant Access (FEX) Discovery

VSS# config t

VSS(config)# interface port-channel 100

VSS(config-if)# switchport

VSS(config-if)# switchport mode fex-fabric

VSS(config-if)# fex associate 110

VSS(config-if)# no shut

VSS(config-if)# exit

VSS(config)# interface range TenGig 1/5/4, TenGig 2/5/4

VSS(config-if)# switchport

VSS(config-if)# channel-group 100 mode on

VSS(config-if)# no shut

FEX Configuration commands

3. Associate it with a fex-id 110

1. Create a Layer 2 Port-channel 100

2. Configure its mode as fex-fabric

4. Select the Fabric physical links

5. Bundle them into using mode on

Repeat the configuration for configuring additional FEX clients






Agenda

Init - First SDP exchange

Connect - Control VLAN and CVIF exchange

Registration - FIN (FEX internal Network) allocation, IDPROM, URI Path

Image Download/Version Mismatch – Happens when version mismatch occurs

Registered - FCP Ready

Online – Once Parent receives FCP Ready from FEX Client, it moves to online state

Offline – Seen when FEX Client is removed or disabled


VSS# show fex

FEX FEX FEX FEX

Number Description State Model Serial

---------------------------------------------------------------------------

110 FEX0110 online C6800IA-48TD FOC1736W1A8

120 FEX0120 online C6800IA-48FPD FOC1736W197

Verify FEX state

FEX States

FEX ids 110 and 120

Current FEX state

FEX model

FEX Serial

Numbers


VSS# show fex detail

FEX: 110 Description: FEX0110 state: online

FEX version: 15.0(2)EX4

Extender Model: C6800IA-48TD, Extender Serial:

FCW1901A4B2

FCP ready: yes

Image Version Check: enforced

Fabric Portchannel Ports: 1

Fabric port for control traffic: Te2/5/4

Fabric interface state:

Po100 - Interface Up.

Te1/5/4 - Interface Up. state: bound


Contd..

Verify detailed FEX status

Contd..

FEX: 120 Description: FEX0120 state: online

FEX version: 15.0(2)EX4

Extender Model: C6800IA-48FPD, Extender Serial:

FOC1736W197

FCP ready: yes

Image Version Check: enforced

Fabric Portchannel Ports: 2

Fabric port for control traffic: Te2/5/3

Fabric interface state:

Po200 - Interface Up.



RSL members are bound and up

FEX is online

Instant Access (FEX) DiscoveryVerify individual FEX members and environment status

VSS#show module fex

Switch Number: 110 Role: FEX

---------------------- -----------------------------

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 48 C6800IA 48GE C6800IA-48TD FCW1901A4B2

2 48 C6800IA 48GE C6800IA-48TD FCW1901A49F

3 48 C6800IA 48GE C6800IA-48TD FCW1901A4AY

4 48 C6800IA 48GE C6800IA-48TD FCW1901A496

<snip>

VSS#show environment status fex all

Fex 110 Module 1

Power-supply 1: Type : Built-in

Status: on

Fex 110 Module 2


Status: on

Fex 110 Module 3


Status: on

<snip>

Fex 110 module 4:

FEX 110 module 4 cooling requirement: 84 cfm

Fex 110 module 1 inlet-1 temperature: 31C



Fex stack number

FEX environment status

FEXswitch model number

Instant Access (FEX) DiscoveryVerify virtual slot-mapVSS#show switch virtual slot-map

Virtual Slot to Remote Switch/Physical Slot Mapping Table:

Virtual Remote Physical Module

Slot No Switch No Slot No Uptime

---------+-----------+----------+----------

17 1 1 -

18 1 2 -

<snip>

50 110 1 00:13:12

51 110 2 00:13:12

52 110 3 00:13:12

53 110 4 00:13:07

54 120 1 00:12:46

55 120 2 00:12:46

56 120 3 00:12:46

57 120 5 00:12:46

<snip>

VSS#test scp ping 50

pinging addr 50(0x32)

assigned sap 0x25

addr 50(0x32) is alive

Switch 1 in FEX 110 stack is mapped to vslot 50

Switch uptime

Testing if Switch 1 in FEX 110 is alive

Instant Access (FEX) DiscoveryControl Plane – Behind the scene

VSS#show platform fex-debug fex 110 sdp

SDP Debug information for FEX : 110

---------------------------------------------

Apr 29 2015 17:26:01 SDP is UP on int TenGigabitEthernet2/5/4

VSS#show monitor event-trace fex clock 17:26

*Apr 29 17:26:00.711: general SDP tx params on intf Te2/5/4

updated with vntag 2049

*Apr 29 17:26:11.923: chassis event FEX 110 ready: state registration

*Apr 29 17:26:11.923: SRP response controller: type:1, ID:110,

FCP:0x2/0x5, IP:192.168.1.1/24, MTS:0x0

*Apr 29 17:26:11.923: SRP response FEX: ID:110, FCP:0x4/0x32, IP:192.16

8.1.110/24, MTS:0x0, image ok:Y

*Apr 29 17:26:12.063: chassis event FEX 110 FCP ready: state registered

*Apr 29 17:26:12.087: chassis event FEX 110 ready: state ready

*Apr 29 17:26:12.087: chassis event FEX 110 online: state online

*Apr 29 17:26:12.091: FCP message FCP ID: FEX 110, num:1, switch:0x4,

slot 50


slot 51


slot 52


slot 53

*Apr 29 17:26:17.487: chassis event FEX 110 FCP ready: state online

SDP state

Switch Discovery Protocol (SDP)

• Fabric Link Discovery

• Switch Discovery

• EtherChannel Link Aggregation

Switch Registration Protocol (SRP)

• Exchange Compatibility information

• IA Client Registration

• IA Client image management

• IA Client OIR

• Stack Member identification & mgmt

Switch Configuration Protocol

• Configuration

• Status

• Statistics

Inter Card Communication (ICC) for Syslog, QoS, Remote login.

Event trace logs for troubleshooting FEX discovery






Agenda

<Interface-type>/<fex-id>/<module>/<submode>/<port>

interface GigabitEthernet 110/1/0/1

Stack Sub Module FEX PortFEX ID

101-199

Catalyst Instant AccessHost Port: Interface Naming Convention

Si Si

interface GigabitEthernet 110/2/0/1

Instant Access (FEX): Interface Naming

VSS#attach fex 110

Attach FEX:110 ip:192.168.1.110

Trying 192.168.1.110 ... Open

ˇ˚ˇ˚ˇ˝ˇ˝

FEX-110>

FEX-110>en

Password: cisco

FEX-110#

FEX-110#show int tenGigabitEthernet 1/0/1

TenGigabitEthernet1/0/1 is up, line protocol is up (connected)

Hardware is Ten Gigabit Ethernet, address is 0022.bdf4.6633 (bia 0022.bdf4.6633)

MTU 9198 bytes, BW 10000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive not set

Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

<snip>

FEX-110#conf t

^

% Invalid input detected at '^' marker.

Remote into a FEX. Password is cisco.

Show commands can be issued from FEX CLI

Configuration mode is not available from FEX CLI. All FEX related configuration needs to be done from parent 6800 configuration mode.

Instant Access (FEX): Console Access

Instant Access (FEX): Platform resource usage

VSS#show fex system platform usage

FEX id usage details

Fex-ids inuse: 110, 120

Fex-ids online: 110, 120

Total Used Free

----- ---- ----

42 2 40

FEX slot usage details

FEX-id Switch-id Vslot Pslot Status

------ --------- ----- ----- ------

110 5 50 1 In-use

110 5 51 2 In-use

110 5 52 3 In-use

110 5 53 4 In-use

120 6 54 1 In-use

120 6 55 2 In-use

120 6 56 3 In-use

120 6 57 5 In-use

Total Used Reserved Temp-Use/Free Free

----- ---- -------- ------------- ----

47 8 0 0/5 39

Current Temp vslot allowed FEXs:

Contd…

Contd…

FEX ports usage details

FEX-id Switch-id Ports

------ --------- -----

110 5 192

120 6 192

Total Used Free

----- ---- ----

2016 384 1632

Stack members usage details

FEX-id Switch-id Used Free

------ --------- ---- ----

110 5 4 1

120 6 4 1

VNTAG MGR Usage

-----------------------

Max unicast VIFs available 2048

Total unicast VIFs used 384

Max non-mdest VIFs available 1019

Total non-mdest VIFs used 2

Max mdest VIFs available 16380

Total mdest VIFs used 0

Total FEX ports supported is 2016 in current releases.

Total FEX switches that can be stacked per stack is 5.

Each FEX module will consume one vslot id

FEX 110 is a stacked FEX with 4 switches bundled into one FEX id

Instant Access (FEX): Adding member to stackVSS#show module fex 110


---------------------- -----------------------------


--- ----- -------------------------------------- ------------------ -----------




%STACKMGR-4-SWITCH_ADDED: Switch 4 has been ADDED to the stack (FEX-110)

%SATMGR-SW1-5-FEX_MODULE_ONLINE: FEX 110, module 4 online

%OIR-SW1-6-INSREM: Switch 110 Physical Slot 4 - Module Type LINE_CARD inserted

%STACKMGR-5-SWITCH_READY: Switch 4 is READY (FEX-110)

%DIAG-SW1-6-RUN_MINIMUM: Fex 110 Module 4: Running Minimal Diagnostics...

%DIAG-SW1-6-DIAG_OK: Fex 110 Module 4: Passed Online Diagnostics

%OIR-SW1-6-SP_INSCARD: Card inserted in Switch_number = 110, physical slot 4, interfaces are now online

VSS#sh module fex 110


---------------------- -----------------------------


--- ----- -------------------------------------- ------------------ -----------




4 48 C6800IA 48GE C6800IA-48TD FCW1901A496

Stack Member automatically

Discovered and associated to

FEX like a Line CardNew member added to stack

Stack before adding 4th member

Instant Access (FEX): Helpful commands

show fex <fex-id> detail

show switch virtual slot-map

test scp ping <vslot id of fex>

show module fex <fex-id>

show platform fex-debug fex <fex-id> sdp

show monitor event-trace fex clock <hh:mm>

show fex image bundle version

debug fex [error|sdp|srp|init]<fex-id>*

debug switch virtual fexmgr[error|event|packet]<fex-id>*

Check in which state FEX is stuck

Get vslot id of fex in question

Check if FEX Internal Network is Up

Check FEX diagnostics have passed

Check if SDP handshake was successful

Event trace utility to store all FEX related events at different stages

Turn on common FEX debugs and restart FEX (run with caution)

Check the version and the stack member details (run with caution)






Agenda

Catalyst Instant AccessVNTAG802.1Qbh

DA[6] SA[6] VNTAG[6] 802.1Q[4] Frame Payload …. CRC[4]

VNTAG ETHER TYPE

(0X8926)D[1] P[1] DVIF [12] L[1] R[1] R[1] R[1] SVIF[12]

Destination

bitPointer bit

(multicast)

Destination VIF Source VIF

Loopback bit Reserved

Unicast D=1 Unicast to FEX Host Port

Multicast P=1 Pointer to Multicast Table on

FEX Client

TAG IA Client Interface

VIF1 IF1

VIF2 IF2

Ingress Mapping

Catalyst Instant AccessLocal Processing of Remote Ports, How?

IA Parent

IA Client

Hosts

Host-1 Host-2

Switch-3

IF1IF2

F101

TAG: Virtual NIC Tag VIF – Virtual Interface

(VIF1) (VIF2)

• Automatically assigned

• One VIF to each host port

• One VIF to each Etherchannel

• One VIF to FEX CPU for Control Channel

• IA Parent VIF = 0

• Multicast/Broadcast: Pointer to Replication

Table in IA Client

96

Catalyst Instant AccessPacket Walk (IA Client Host Port to IA Parent)

IA Parent

IA Client

Host-1

Switch-3

IF1

F101

VNTAG: Virtual NIC Tag VIF – Virtual Interface

(VIF1)

MAC + Payload

VNTAG

SVIF = VIF1 DVIF = 0

MAC + Payload

MAC + Payload

VNTAG


MAC + Payload

Catalyst Instant AccessPacket Walk (IA Parent to IA Client Host Port)

IA Parent

IA Client

Host-1

Switch-3

IF1

F101


(VIF1)

MAC + Payload

VNTAG

SVIF = 0, DVIF = VIF1

MAC + Payload

MAC + Payload

VNTAG


MAC + Payload

Catalyst Instant AccessPacket Walk (Host 1 to Host 2)

Host-1

MAC1

Host-2

MAC2

Switch-3IF2

F101

VNTAG: Virtual NIC Tag

(VIF2)

SA=MAC1, DA=MAC2+ Payload

IF1

(VIF1)


VNTAG



VNTAG




SA=MAC1, DA=MAC2+ PayloadSA=MAC1, DA=MAC2+ Payload

VNTAG


VNTAG




Catalyst Instant AccessPacket Walk - Multicast / Broadcast

IA Parent

IA Client

Hosts

Host-1 Host-2

Switch-3

IF1IF2

F101


(VIF1) (VIF2)

MAC + Payload

VNTAG, P=1

SVIF =0, DVIF = Group VIF

MAC + Payload

Outgoing Interface

IF1, IF2

192.168.1.100, 224.0.255.1Incoming Interface: FortyGig 5/1 RPF Neighbor 210.20.37.33 Outgoing interface list:

Gigabitethernet 101/1/0/1, Forward/Dense, 0:57:31/0:02:52Gigabitethernet 101/1/0/2, Forward/Dense, 0:56:55/0:01:28

MAC + Payload

Group VIF

MAC + Payload

VSS#show run interface g110/1/0/48

interface GigabitEthernet110/1/0/48

switchport

switchport trunk allowed vlan 1

switchport mode access

VSS#show mac address-table interface g110/1/0/48 all detail

Displaying entries from active supervisor:

b line pg: bd mac-address st gm pi cap sec rm rma m nf tr al ag t s fl index

-+----+--+-----+--------------+--+--+--+---+---+--+---+-+--+--+--+--+-+-+--+-------

0 D94 0 1 0000.0200.0300 0 0 0 0 0 0 0 1 0 0 0 49 0 0 0 0x206F

Displaying entries from DFC switch [1] linecard [2]:

b line pg: bd mac-address st gm pi cap sec rm rma m nf tr al ag t s fl index

-+----+--+-----+--------------+--+--+--+---+---+--+---+-+--+--+--+--+-+-+--+-------

0 D94 0 1 0000.0200.0300 0 0 0 0 0 0 0 1 0 0 0 45 0 0 0 0x206F

VSS#test platform software switch virtual vntag_mgr vif-map LTL 0x206F detail

VIF INFO:

VIF# 112Type UNICAST VIF

LTL# 206FOperStatus# 2

Detail option is hidden

LTL index associated with MAC address

FEX port connecting host

VIF used to send packet to FEX

VIF is in hardware

0 Status Unavailable

1 programming pending

2 programming success

4 programming failure

8 MCAST CB Pending

16 Delete pending

Catalyst Instant AccessVerifying LTL to VIF map – from VSS

VSS#attach fex 110

FEX-101>en

FEX-110#show platform fex ucast-entries | include 112

vif sw_idb portname GPN handle res_index

==== ========== ====================== ==== ========= =========

112 0x7710E04 GigabitEthernet1/0/48 48 0x34 0x3980000

FEX-110#

FEX-110#sh platform pm if-numbers | include 1/0/48

interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb

----------------------------------------------------------------------

Gi1/0/48 48 48 48 1/25 1 48 48 local Yes Yes

FEX-110#sh platform port-asic stats drop port 25 asic 1

Port-asic Port Drop Statistics - Summary

========================================

Port 0 TxQueue Drop Stats: 0

Port 1 TxQueue Drop Stats: 0

<snip>

Queue 0

Weight 0 Frames 0

Weight 1 Frames 0

<snip>

Catalyst Instant AccessVerifying LTL to VIF map- from 6800IA

Host port mapped to VIF 122

Check hardware level drops on port 25 on asic 1 on 6800IA

6800IA port-asic mapped to host port

Troubleshooting Catalyst 6500/6800 Switches Final Message

Please practice and get familiar with the troubleshooting techniques.

If you don’t use it, you lose it.

Catalyst 6500/6800 is thriving ….. and …..

Innovation Continues !!!

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle shashankcisco

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could Be a Winner

http://bit.ly/CLUSwin

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you

Troubleshooting Cisco Catalystd2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/BRKCRS-3143.pdf ·...

Documents

Transcript of Troubleshooting Cisco Catalystd2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/BRKCRS-3143.pdf ·...