Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP...

510

Click here to load reader

Transcript of Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP...

Page 1: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Trouble Shooting

CERNET NOCPengfei Li

Page 2: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

2

Contents

Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting CEF Trouble Shooting IP Multicast Case Studies

Page 3: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Trouble Shooting Tools

Key of trouble shooting –Excellent NMS

Page 4: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

4

Route Monitoring

Page 5: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

5

Routing (BGP summary)

Page 6: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

6

Routing Mornitoring

Page 7: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

7

BGP Statistics (current status)

Page 8: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

8

Looking-glass

Page 9: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

9

BGP Monitoring (TEIN2-NORTH)

Page 10: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

10

BGP Monitoring (TEIN2-SOUTH)

Page 11: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

11

BGP Monitoring (TEIN2-JP)

Page 12: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

12

AS Path Entries

Page 13: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

13

Community Entries

Page 14: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

14

IPv4 Prefix

Page 15: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

15

IPv6 Prefix

Page 16: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

16

Basic tools

Ping Traceroute telnet Show commands etc

Page 17: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Trouble Shooting BGP

Page 18: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

18

BGP in Large Scale Networks

Page 19: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

19

Avoid the Problem in theFirst Place Use simple configurations maintain a consistent policy througho

ut the AS Promote stable networks nail-down your routes use loopback in

terfaces Grow into your network use peer-groups and RRs for scalability

Page 20: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

20

Agenda

Basic Tools Peer Establishment UPDATE Exchange Selection Algorithm Route Reflectors Route Flap Damping

Page 21: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

21

BGP Troubleshooting Tools

show commands debug output Log messages

Page 22: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

22

show Commandsrouter#show ip bgp ? A.B.C.D IP prefix <network>/<length>, e.g., 35.0.0.0/8 A.B.C.D Network in the BGP routing table to display cidr-only Display only routes with non-natural netmasks community Display routes matching the communities community-list Display routes matching the community-list dampened-paths Display paths suppressed due to dampening filter-list Display routes conforming to the filter-list flap-statistics Display flap statistics of routes inconsistent-as Display only routes with inconsistent origin ASs neighbors Detailed information on TCP and BGP neighbor connections paths Path information peer-group Display information on peer-groups quote-regexp Display routes matching the AS path "regular expression" regexp Display routes matching the AS path regular expression summary Summary of BGP neighbor status | Output modifiers <cr>

Page 23: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

23

show Commands (Cont.)

router#show ip bgp neighbors x.x.x.x ? advertised-routes Display the routes advertised to a BGP neighbor dampened-routes Display the dampened routes received from neighbor flap-statistics Display flap statistics of the routes learned from

neighbor paths Display AS paths learned from neighbor received Display information received from a BGP neighbor received-routes Display the received routes from neighbor routes Display routes learned from neighbor | Output modifiers <cr>

Page 24: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

24

router#show ip bgp BGP table version is 9, local router ID is 7.72.6.1 Status codes: s suppressed, d damped, h history, * valid, > best, i -

internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 3.0.0.0 0.0.0.0 0 32768 i *> 5.0.0.0 0.0.0.0 0 32768 i *> 6.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 7.0.0.0 0.0.0.0 0 32768 i *> 8.0.0.0/5 0.0.0.0 0 32768 i *> 17.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 23.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i *> 35.0.0.0 6.72.6.2 4294967294 0 2 i * i 6.72.6.2 4294967294 100 0 2 i

The BGP Table

Page 25: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

25

The BGP Table (Cont.)

router#show ip bgp 6.0.0.0 BGP routing table entry for 6.0.0.0/8, version 2Paths: (2 available, best #1) Advertised to non peer-group peers: 7.25.14.4 7.72.6.3 7.75.7.1 2 6.72.6.2 from 6.72.6.2 (7.72.6.2) Origin IGP, metric 4294967294, localpref 100, valid, external, best 2 6.72.6.2 from 7.75.7.1 (7.75.7.1) Origin IGP, metric 4294967294, localpref 100, valid, internal

Page 26: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

26

show ip bgp Summary

router#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1BGP table version is 9, main routing table version 98 network entries and 12 paths using 1176 bytes of memory3 BGP path attribute entries using 144 bytes of memory1 BGP AS-PATH entries using 24 bytes of memoryBGP activity 8/0 prefixes, 12/0 paths

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 6885 6882 9 0 0 4d18h 47.25.14.4 4 3 6882 6883 9 0 0 4d18h 07.72.6.3 4 1 6880 6886 9 0 0 4d18h 07.75.7.1 4 1 6884 6885 9 0 0 4d18h 4

Page 27: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

27

show ip bgp neighborsrouter#show ip bgp neighbors 6.72.6.2BGP neighbor is 6.72.6.2, remote AS 2, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 7.72.6.2 BGP state = Established, table version = 9, up for 4d21h Last read 00:00:56, last send 00:00:48 Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Peer negotiated unicast and multicast routes Exchanging unicast routes only Received route refresh capability from peer Minimum time between advertisement runs is 30 seconds Received 7044 messages, 0 notifications, 0 in queue Sent 7041 messages, 0 notifications, 0 in queue Prefix advertised 4, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Inbound path policy configured Route map for incoming advertisements is k Connections established 1; dropped 0 Last reset never Number of unicast/multicast prefixes received 4/0 External BGP neighbor may be up to 255 hops away.Connection state is ESTAB, I/O status: 1, unread input bytes: 0Local host: 3.72.6.1, Local port: 179Foreign host: 6.72.6.2, Foreign port: 11014

Page 28: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

28

router#debug ip bgp ? A.B.C.D BGP neighbor address dampeningBGP dampening events BGP events keepalives BGP keepalives updates BGP updates <cr>

debug ip bgp

Remember—can be dangerous! Use only in the lab or If advised by the TAC

To make a little safer: logging buffered <size> no logging console

Page 29: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

29

Session Establishment (debug ip bgp )

16:06:30: BGP: 7.72.6.1 sending OPEN, version 416:06:31: BGP: 7.72.6.1 OPEN rcvd, version 416:06:31: BGP: 7.72.6.1 rcv OPEN w/ OPTION parameter len: 1216:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 616:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 1, length 416:06:31: BGP: 7.72.6.1 OPEN has MP_EXT CAP for afi/safi: 1/116:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 216:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 128, length 0

16:06:31: BGP: 7.75.7.1 passive open16:06:31: BGP: 7.75.7.1 OPEN rcvd, version 416:06:31: BGP: 7.75.7.1 sending OPEN, version 416:06:31: BGP: 7.75.7.1 rcv OPEN w/ OPTION parameter len: 1216:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 616:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 1, length 416:06:31: BGP: 7.75.7.1 OPEN has MP_EXT CAP for afi/safi: 1/116:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 216:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 128, length 0

Page 30: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

30

Session Establishment(debug ip bgp events)

17:31:39: BGP: 7.72.6.1 went from Idle to Active17:32:00: BGP: 7.72.6.1 went from Active to OpenSent17:32:00: BGP: 7.72.6.1 went from OpenSent to OpenConfirm17:32:00: BGP: 7.72.6.1 went from OpenConfirm to Established

17:31:59: BGP: 7.75.7.1 went from Idle to Active17:32:00: BGP: 7.75.7.1 went from Active to Idle17:32:00: BGP: 7.75.7.1 went from Idle to Connect17:32:00: BGP: 7.75.7.1 went from Connect to OpenSent17:32:00: BGP: 7.75.7.1 went from OpenSent to OpenConfirm17:32:00: BGP: 7.75.7.1 went from OpenConfirm to Established

Page 31: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

31

Looking at the Updates

router#debug ip bgp updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr>

router#debug ip bgp x.x.x.x updates? <1-199> Access list <1300-2699> Access list (expanded range) <cr>

Use an access-list to limit the output!Use an access-list to limit the output!

Page 32: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

32

debug ip bgp Updates

BGP: 6.72.6.2 computing updates, neighbor version 0, table version 13, starting at 0.0.0.0BGP: 6.72.6.2 send UPDATE 3.0.0.0/8, next 3.72.6.1BGP: , metric 0, path 1BGP: 6.72.6.2 send UPDATE 5.0.0.0/8 (chgflags: 0x0), next 3.72.6.1BGP: 6.72.6.2 send UPDATE 7.0.0.0/8 (chgflags: 0x0), next 3.72.6.1BGP: 6.72.6.2 1 updates enqueued (average=56, maximum=56)BGP: 6.72.6.2 update run completed, ran for 0ms, neighbor version 0, start version 13, throttled to 13, check point net 0.0.0.0

Peer Address Prefix Being Advertised NEXT_HOP

Page 33: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

33

debug ip bgp Updates (Cont.)

BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric 294, path 2 1BGP: 6.72.6.2 rcv UPDATE about 3.0.0.0/8 -- DENIED due to: as-path contains our own AS;BGP: 6.72.6.2 rcv UPDATE about 7.0.0.0/8 -- DENIED due to: as-path

contains our own AS;

BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric 494, path 2BGP: 6.72.6.2 rcv UPDATE about 6.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 17.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 23.0.0.0/8BGP: 6.72.6.2 rcv UPDATE about 35.0.0.0/8

Prefixes in the Same UPDATE

Attributes Apply to

All Prefixes

Peer Address

Page 34: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

34

Logging Neighbor Changes

Generate a log message whenever a BGP neighbor changes state, also indicate reason for reset

Syntax (router subcommand): [no] bgp log-neighbor-changes

Typical log messages: %BGP-5-ADJCHANGE: neighbor x.x.x.x Up %BGP-5-ADJCHANGE: neighbor x.x.x.x Down-Remote AS

changed

Page 35: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

35

show ip bgp neighbors x.x.x.x

router#show ip bgp neighbors 7.75.7.1BGP neighbor is 7.75.7.1, remote AS 2, external link...Received 194 messages, 1 notifications, 0 in queue Sent 194 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 7; dropped 7 Last reset 00:04:11, due to BGP Notification received, hold time expired Number of unicast/multicast prefixes received 0/0 External BGP neighbor may be up to 255 hops away. No active TCP connection

Page 36: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

36

show ip bgp neighbors x.x.x.x cont. "BGP protocol initialization" "No memory" "BGP Notification received" "BGP Notification sent" "User reset" "Peer timeout” "Password change” "Error during connection collision" "Peer closed the session" "Peer over prefix limit" "Interface flap" "Router ID changed” "Neighbor deleted" "Member added to peergroup" "Admin. shutdown" "Remote AS changed" "Capability changed" "RR client config change” "Soft reconfig change" "Local AS change"

Page 37: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

37

Common Problems

Sessions are not establishedNo IP reachabilityIncorrect configuration

Peering addressesOPEN parameters

Page 38: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

38

Can’t Establish Session - Symptoms

The peering session is not established! State may change between active,

idle and connect

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1 BGP table version is 4, main routing table version 4 6 network entries and 6 paths using 774 bytes of memory 2 BGP path attribute entries using 96 bytes of memory 1 BGP AS-PATH entries using 24 bytes of memory BGP activity 6/0 prefixes, 6/0 paths

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 0 0 0 0 0 never Idle7.25.14.4 4 3 4 5 4 0 0 00:01:43 07.72.6.3 4 1 0 0 0 0 0 never Active7.75.7.1 4 1 7 5 4 0 0 00:01:55 3

Page 39: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

39

The BGP Finite State Machine has 5 states:

Idle: no resources are allocated Connect: waiting for TCP session to be establish

ed; no connection is established actively, but one may be accepted

Active: actively trying to establish a TCP connection

OpenSent: waiting for OPEN message from peer OpenConfirm: waiting for confirmation of the O

PEN message (KEEPALIVE or NOTIFICATION)Established: UPDATES may now be exchanged…t

his is the normal operational state of BGP.

Page 40: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

40

Can’t Establish Session— Troubleshooting I

Is the remote-as assigned correctly?

router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 7.72.6.3 remote-as 1

Local AS eBGP Peer

iBGP Peer

Page 41: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

41

Can’t Establish Session— Troubleshooting I (Cont.)

Verify IP connectivity check the routing table use ping/trace to verify two way reachability inspect for ACLs in the path to the neighbor

routerA#show ip route 7.72.6.3Routing entry for 7.72.6.3/32 Known via "ospf 123”, distance 110, metric 87, type intra area Last update from 27.27.27.254 on POS5/0, 00:09:33 ago Routing Descriptor Blocks: * 27.27.27.254, from 7.72.6.3, 00:09:33 ago, via POS5/0 Route metric is 87, traffic share count is 1

routerA#ping 7.72.6.3Sending 5, 100-byte ICMP Echos to 7.72.6.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 28/30/32 ms

Page 42: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

42

Can’t Establish Session— Troubleshooting I (Cont.)

routerA#debug ip bgpBGP debugging is on10:51:02: BGP: 7.72.6.3 open active, delay 6864ms10:51:09: BGP: 7.72.6.3 open active, local address 27.27.27.25310:51:09: BGP: 7.72.6.3 open failed: Connection refused by remote host

• Is the remote router configured for BGP?What IP address is the remote router configured to receive?

router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.1 remote-as 1

Page 43: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

43

The TCP session is alwaysThe TCP session is alwayssourced from thesourced from the closestclosest IP IP address to the destination!address to the destination!

Can’t Establish Session— Troubleshooting I (Cont.)Can’t Establish Session— Troubleshooting I (Cont.)

Configuration:Router Arouter bgp 1neighbor 27.27.27.254 remote-as 1Router Crouter bgp 1neighbor 27.27.27.253 remote-as 1

A C

27.27.27.254

27.27.27.253

If redundant paths exist, If redundant paths exist, useuse loopback interfacesloopback interfaces to to

establish the session.establish the session.

Page 44: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

44

Can’t Establish Session— Troubleshooting I (Cont.)

Solution: make sure both routers source the

information from the appropriate interface

routerA#debug ip tcp transactions11:19:48: BGP: 7.72.6.3 open active, delay 9916ms11:19:53: TCP: sending RST, seq 0, ack 309812912111:19:53: TCP: sent RST to 7.7.7.6:11719 from 7.72.6.1:179

router bgp 1 neighbor 7.72.6.3 remote-as 1 neighbor 7.72.6.3 update-source Loopback0

Information sourcedInformation sourcedfrom the IP address in from the IP address in interface Loopback0interface Loopback0

Page 45: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

45

Can’t Establish Session—Symptoms

The eBGP session is still having trouble!

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1BGP table version is 4, main routing table version 46 network entries and 6 paths using 774 bytes of memory2 BGP path attribute entries using 96 bytes of memory1 BGP AS-PATH entries using 24 bytes of memoryBGP activity 6/0 prefixes, 6/0 paths

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 0 0 0 0 0 never Idle7.25.14.4 4 3 385 385 4 0 0 06:22:17 07.72.6.3 4 1 42 49 4 0 0 00:00:15 07.75.7.1 4 1 388 385 4 0 0 06:22:30 3

Page 46: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

46

routerA#configure terminal Enter configuration commands, one per line. End with CNTL/Z.routerA(config)#ip route 6.72.6.2 255.255.255.255 1.1.1.5

routerA#ping 6.72.6.2Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 6.72.6.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Can’t Establish Session - Troubleshooting II

Verify IP connectivity check the routing table use ping/trace to verify two way

reachabilityrouterA#show ip route 6.72.6.2%Network not in table

Page 47: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

47

Can’t Establish Session— Troubleshooting II (Cont.)

Peering with a loopback interface Advantages

Interface is always up Multiple physical paths may

exist to reach it Disadvantages

Physical link failure may take longer to detect

Page 48: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

48

Can’t Establish Session— Troubleshooting II (Cont.)

The debug output indicates the neighbor’s configured peering address

routerA#debug ip bgprouterA#debug ip tcp transactions13:25:30: TCP: sending RST, seq 0, ack 203010066913:25:30: TCP: sent RST to 6.72.6.2:11041 from 3.72.6.1:179

Neighbor is tryingNeighbor is tryingto peer with this IPto peer with this IP

addressaddress

router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 update-source Loopback1

Page 49: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

49

Can’t Establish Session— Troubleshooting II (Cont.)

Hint: by default, eBGP peers should be directly connected in this case, the peering address

doesn’t match a connected interface in the local router

13:33:30: TCP: sending RST, seq 0, ack 251012964513:33:30: TCP: sent RST to 6.72.6.2:11045 from 3.72.6.1:179

Page 50: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

50

Can’t Establish Session— Troubleshooting II (Cont.)

routerA#show ip bgp neighbors 6.72.6.2BGP neighbor is 6.72.6.2, remote AS 2, external link Index 1, Offset 0, Mask 0x2 BGP version 4, remote router ID 0.0.0.0 BGP state = Idle, table version = 0 Last read 00:00:06, last send never Hold time 180, keepalive interval 60 seconds Neighbor NLRI negotiation: Configured for unicast routes only Minimum time between advertisement runs is 30 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Prefix advertised 0, suppressed 0, withdrawn 0 Route refresh request: received 0, sent 0 Connections established 0; dropped 0 Last reset never Number of unicast/multicast prefixes received 0/0 External BGP neighbor not directly connected. No active TCP connection

Page 51: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

51

Can’t Establish Session— Troubleshooting II (Cont.)

At this point, the session should come up

router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 ebgp-multihop 255 neighbor 6.72.6.2 update-source Loopback1

Page 52: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

52

Can’t Establish Session— Symptoms

Still having trouble! Connectivity issues have already been checked and corrected.

routerA#show ip bgp summary BGP router identifier 7.72.6.1, local AS number 1… Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd6.72.6.2 4 2 10 26 0 0 0 never Active

router bgp 1 neighbor 6.72.6.2 remote-as 2 neighbor 6.72.6.2 ebgp-multihop 255 neighbor 6.72.6.2 update-source Loopback1

Page 53: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

53

Can’t Establish Session— Troubleshooting II (Cont.)

If an error is detected, a notification is sent and the session is closed In this case the remote router had

a bad configuration

14:06:37: BGP: 6.72.6.2 open active, local address 3.72.6.114:06:37: BGP: 6.72.6.2 went from Active to OpenSent14:06:37: BGP: 6.72.6.2 sending OPEN, version 414:06:37: BGP: 6.72.6.2 received NOTIFICATION 2/2

(peer in wrong AS) 2 bytes 000114:06:37: BGP: 6.72.6.2 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 6.72.6.2 went from OpenSent to Idle14:06:37: BGP: 6.72.6.2 closing

Page 54: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

54

UPDATE Exchange

Once the session has been established, UPDATEs are exchanged all the locally known routes only the bestpath is advertised

Incremental UPDATE messages are exchanged afterwards

Page 55: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

55

Propagation Decisions

bestpath received from eBGP peer advertise to all peers

bestpath received from iBGP peer advertise only to eBGP peers a full iBGP mesh must exist

Page 56: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

56

Common Problems

Missing routes No iBGP full mesh Filters: routes are not received/sent

Slow convergence

Page 57: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

57

Missing Routes—Troubleshooting Steps

Determine which filters are applied

to the BGP session show ip bgp neighbors x.x.x.x Look at the configuration

Examine the route and pick out the relevant attributes show ip bgp x.x.x.x

Page 58: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

58

Missing Routes—Troubleshooting Steps (Cont.)

Compare the route against the filters

If no match is found Use route-refresh or soft-

reconfiguration Filter the updates through an ACL to

determine where the problem is

Page 59: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

59

Missing Routes—Symptoms

Missing 4.0.0.0/8 in 7.75.7.1 (routerA) not received from 7.72.6.3 (routerB)

routerB#sh ip bgp nei 7.75.7.1 advertised-routes | include 4.0.0.0*> 4.0.0.0 0.0.0.0 0 32768 i

routerB shows that the route was advertised to routerA!

Page 60: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

60

Missing Routes—Troubleshooting

routerA#show access-lists 10Standard IP access list 10 permit 4.0.0.0

routerA#debug ip bgp 7.72.6.3 updates 10BGP updates debugging is on for access list 10 for neighbor 7.72.6.3

routerA#clear ip bgp 7.72.6.3 in01:22:41: BGP: 7.72.6.3 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i, metric 0, path 201:22:41: BGP: 7.72.6.3 rcv UPDATE about 4.0.0.0/8 -- DENIED due

to: distribute/prefix-list;

Page 61: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

61

Missing Routes—Troubleshooting (Cont.)

router bgp 1 no synchronization bgp log-neighbor-changes neighbor 7.72.6.3 remote-as 2 neighbor 7.72.6.3 ebgp-multihop 255 neighbor 7.72.6.3 update-source Loopback0 neighbor 7.72.6.3 prefix-list filter in!ip prefix-list filter seq 5 deny 4.0.0.0/8ip prefix-list filter seq 10 permit 0.0.0.0/0 le 32

Page 62: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

62

Common Problems

Inconsistent decision/policy MED External paths Communities By default, communities are not

propagated neighbor x.x.x.x send-community

Page 63: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

63

Inconsistent Decision—Symptom I

The bestpath changes every time the peering is reset.

routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 40Paths: (3 available, best #3, advertised over IBGP, EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external, best

Page 64: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

64

Inconsistent Decision—Symptom I (Cont.)

routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 2Paths: (3 available, best #3, advertised over EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal, best

Same paths, but different result!

Page 65: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

65

Inconsistent Decision—Symptom I (Cont.)

Different result…again!!

routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 12Paths: (3 available, best #3, advertised over EBGP) 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best

Page 66: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

66

Deterministic MED

By default, the prefixes are compared in order of arrival it may result in inconsistent decisions use bgp deterministic-med the bestpath is recalculated as soon

as the command is entered enable in all the routers in the AS

Page 67: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

67

Deterministic MED—Operation

The paths are ordered by peer-AS The bestpath for each group

is selected The overall bestpath results from

comparing the winners in each group

Page 68: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

68

Deterministic MED—Result

The bestpath will always be the same!

routerA#sh ip bgp 160.100.0.0BGP routing table entry for 160.100.0.0/16, version 15Paths: (3 available, best #1, advertised over EBGP) 1 204.146.33.10 from 204.146.33.10 (204.146.33.1) Origin IGP, metric 0, localpref 100, valid, internal, best 3 204.146.33.66 from 204.146.33.66 (204.146.33.2) Origin IGP, metric 20, localpref 100, valid, internal 3 204.146.33.6 from 204.146.33.6 (10.4.1.1) Origin IGP, metric 30, valid, external

Page 69: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

69

Inconsistent Decision—Symptom II

The bestpath changes every time the peering is reset

routerA#show ip bgp 7.0.0.0BGP routing table entry for 7.0.0.0/8, version 15Paths: (2 available, best #2) Not advertised to any peer 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external, best

Page 70: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

70

Inconsistent Decision—Symptom II (Cont.)

The “oldest” external is the bestpath. All other attributes are the same Stability enhancement!

routerA#show ip bgp 7.0.0.0 BGP routing table entry for 7.0.0.0/8, version 17Paths: (2 available, best #2) Not advertised to any peer 2 21.21.21.254 from 21.21.21.254 (7.75.7.1) Origin IGP, metric 0, localpref 100, valid, external 2 1.1.1.5 from 1.1.1.5 (1.1.1.1) Origin IGP, metric 0, localpref 100, valid, external, best

Page 71: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

71

Routes Flapping BGP learned routes change periodically res

ulting in high cpu BGP prefixes bounces between a suboptim

al path through a different provider and a recursive route

Recursive routing occurs when the next hop address of a network is an address of that network itself

Page 72: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

72

Routes Flapping (Contd..)

Page 73: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

73

Routes Flapping (Contd..) The DMZ address space between AS1

and AS3 (161.108.0.0/16) also belongs to AS3. The address 161.108.0.3 is the next hop address advertised by AS3-EBGP toward AS1-IBGP.

Router AS1-IBGP is NOT advertising the DMZ address space into AS1 via the IGP. This in combination with not having next-hop-self configured

Page 74: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

74

Routes Flapping (Contd..)

S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 0 Paths: (1 available, no best path) 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal

Page 75: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

75

Routes Flapping (Contd..)

S2-R3-2500#show ip bgp 161.108.0.0 BGP routing table entry for 161.108.0.0/16, version 0 Paths: (1 available, no best path) 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal

Page 76: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

76

Routes Flapping (Contd..)

After EBGP session comes upS2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 4 Paths: (2 available, best #1, advertised over IBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external, best 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal S2-R3-2500#

Page 77: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

77

Routes Flapping (Contd..)

The next time that BGP scans the routing table, we have a route to thenext hop 161.108.0.3 via AS2 S2-R3-2500# BGP: scanning routing tables RT: del 161.108.0.0 via 131.108.50.1, bgp metric [20/0] RT: delete network route to 161.108.0.0 RT: add 161.108.0.0/16 via 161.108.0.3, bgp metric [200/0] RT: del 192.168.1.0 via 131.108.50.1, bgp metric [20/0] RT: delete network route to 192.168.1.0 RT: add 192.168.1.0/24 via 161.108.0.3, bgp metric [200/0]

Page 78: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

78

Routes Flapping (Contd..)

These routes, once installed This inconsistency is discovered the next time that BGP computes an update for his AS2 neighbor, are recursive.

S2-R3-2500# BGP: 131.108.50.1 computing updates, neighbor version 2, table

version 10, starting at 0.0.0.0 RT: recursion error routing 161.108.0.3 - probable routing loop RT: recursion error routing 161.108.0.3 - probable routing loop BGP: 131.108.50.1 update run completed, ran for 8ms, neighbor

version 2, start version 10, throttled to 10, check point net 0.0.0.0

Page 79: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

79

Routes Flapping (Contd..)

These unreachable routes will be marked appropiately the next time BGP scans the routing table and will be removed.

S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 10 Paths: (2 available, best #2, advertised over IBGP, EBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external 3 161.108.0.3 from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal, best

Page 80: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

80

Routes Flapping (Contd..)

We can see that the routes are changed and the routes via the next hop address of 161.108.0.3 are removed the next time that BGP scans:

BGP: scanning routing tables RT: del 161.108.0.0 via 161.108.0.3, bgp metric [200/0] RT: delete network route to 161.108.0.0 RT: add 161.108.0.0/16 via 131.108.50.1, bgp metric [20/0] RT: del 192.168.1.0 via 161.108.0.3, bgp metric [200/0] RT: delete network route to 192.168.1.0 RT: add 192.168.1.0/24 via 131.108.50.1, bgp metric [20/0] S2-R3-2500#

Page 81: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

81

Routes Flapping (Contd..)

S2-R3-2500#show ip bgp 192.168.1.0 BGP routing table entry for 192.168.1.0/24, version 12 Paths: (2 available, best #1, advertised over IBGP, EBGP) 2 3 131.108.50.1 from 131.108.50.1 (10.1.1.1) Origin IGP, localpref 100, valid, external, best 3 161.108.0.3 (inaccessible) from 172.108.10.1 Origin IGP, metric 0, localpref 100, valid, internal

Page 82: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

82

BGP Case Studies

© 2002, Cisco Systems, Inc. All rights reserved. 82

Page 83: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case 1 : Peer Establishment

Missing RoutesInconsistent Route Selection

Loops and Convergence IssuesAdd CERNET CASE

Page 84: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

84

Peer Establishment

Routers establish a TCP session Port 179—permit in ACLs IP connectivity (route from IGP)

OPEN messages are exchanged Peering addresses must match the

TCP session Local AS configuration parameters

Page 85: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

85

Common Problems

Sessions are not established No IP reachability Incorrect configuration

Peers are flapping Layer 2 problems

Page 86: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

86

Peer Establishment—Diagram

R2#sh run | begin bgp router bgp 1 bgp log-neighbor-changes Neighbor 1.1.1.1 remote-as 1 Neighbor 3.3.3.3 remote-as 2

AS 1

AS 2

R1R1

IBGPEBGP

1.1.1.1 2.2.2.2

3.3.3.3?

?

R2R2

R3R3

Page 87: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

87

R2#show ip bgp summary

BGP router identifier 2.2.2.2, local AS number 1

BGP table version is 1, main routing table version 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State

1.1.1.1 4 1 0 0 0 0 0 never Active

3.3.3.3 4 2 0 0 0 0 0 never Idle

Peer Establishment—Symptoms

Both peers are having problems State may change between active,

idle and connect

Page 88: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

88

Peer Establishment

Is the remote-as assigned correctly? Verify with your diagram or other

documentation!

R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 3.3.3.3 remote-as 2

Local AS

eBGP Peer

iBGP Peer

Page 89: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

89

Peer Establishment—IBGP

Assume that IP connectivity has been checked Check TCP to find out what connections we are accepting

R2#show tcp brief allTCB Local Address Foreign Address (state)005F2934 *.179 3.3.3.3.* LISTEN0063F3D4 *.179 1.1.1.1.* LISTEN

We are listening for TCP connections for port 179 for the configured peering addresses only!

R2#debug ip tcp transactionsTCP special event debugging is onR2#TCP: sending RST, seq 0, ack 2500483296TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179

Remote is trying to open the session from 4.4.4.4 address …

Page 90: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

90

Peer Establishment—IBGPWhat about us ?

R2#debug ip bgp BGP debugging is onR2#BGP: 1.1.1.1 open active, local address 4.4.4.5BGP: 1.1.1.1 open failed: Connection refused by remote host

We are trying to open the session from 4.4.4.5 address…

R2#sh ip route 1.1.1.1Routing entry for 1.1.1.1/32 Known via "static", distance 1, metric 0 (connected) * directly connected, via Serial1 Route metric is 0, traffic share count is 1

R2#show ip interface brief | include Serial1Serial1 4.4.4.5 YES manual up up

Page 91: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

91

Peer Establishment—IBGP

Source address is the outgoing interface towards the destination but peering in this case is using loopback interfaces!

Force both routers to source from thecorrect interface

Use “update-source” to specify the loopback when loopback peering

R2#router bgp 1 neighbor 1.1.1.1 remote-as 1 neighbor 1.1.1.1 update-source Loopback0 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 update-source Loopback0

Page 92: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

92

Peer Establishment—Symptoms

R1 is established now The EBGP session is still having

trouble!

R2# sh ip bgp summaryBGP router identifier 2.2.2.2, local AS number 1BGP table version is 1, main routing table version 1Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 1 7 7 1 0 0 00:00:24 33.3.3.3 4 2 0 0 0 0 0 never Idle

Page 93: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

93

R2#ping 3.3.3.3Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

Peer Establishment—EBGP Trying to load-balance over multiple links to the eBGP

peer Verify IP connectivity

Check the routing table Use ping/trace to verify two way

reachability

Routing Towards Destination Correct, but…

Page 94: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

94

Peer Establishment—EBGPR2# ping ipTarget IP address: 3.3.3.3Repeat count [5]: Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: ySource address or interface: 2.2.2.2Type of service [0]: Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]: Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:.....Success rate is 0 percent (0/5)

No Route Back from our Peer!

Page 95: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

95

Peer Establishment—EBGP

R2#sh ip bgp neigh 3.3.3.3BGP neighbor is 3.3.3.3, remote AS 2, external link BGP version 4, remote router ID 0.0.0.0 BGP state = Idle Last read 00:00:04, hold time is 180, keepalive interval is 60 seconds Received 0 messages, 0 notifications, 0 in queue Sent 0 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Default minimum time between advertisement runs is 30 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0 Index 2, Offset 0, Mask 0x4 0 accepted prefixes consume 0 bytes Prefix advertised 0, suppressed 0, withdrawn 0 Connections established 0; dropped 0 Last reset never External BGP neighbor not directly connected. No active TCP connection

Neighbor Added Route but Still Having Problems?…

Page 96: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

96

Peer Establishment—EBGP

eBGP peers are normally directly connected By default, TTL is set to 1 for eBGP peers If not directly connected, specify ebgp-multihop

At this point, the session should come up

router bgp 1 neighbor 3.3.3.3 remote-as 2 neighbor 3.3.3.3 ebgp-multihop 255 neighbor 3.3.3.3 update-source Loopback0

Page 97: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

97

Peer Establishment—EBGP

Still having trouble! Connectivity issues have already

been checked and corrected

R2#show ip bgp summary BGP router identifier 2.2.2.2, local AS number 1 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd3.3.3.3 4 2 10 26 0 0 0 never Active

Page 98: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

98

Peer Establishment—EBGP

If an error is detected, a notification is sent and the sessionis closed

R3 is configured incorrectly Has “neighbor 2.2.2.2 remote-as 10” Should have “neighbor 2.2.2.2 remote-as 1”

After R3 makes this correction the session comes up

R2#debug ip bgp events14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.214:06:37: BGP: 3.3.3.3 went from Active to OpenSent14:06:37: BGP: 3.3.3.3 sending OPEN, version 414:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2

(peer in wrong AS) 2 bytes 000114:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT14:06:37: BGP: service reset requests14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle14:06:37: BGP: 3.3.3.3 closing

Page 99: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case 2 : Load Balancing

Parallel Links eBGP Multipath

Multi-homed

Page 100: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

100

Command

neighbor {ip-address | peer-group-name} ebgp-multihop [ttl]

To accept and attempt BGP connections to external peers residing on networks that are not directly connected, use the neighbor ebgp-multihop command in router configuration mode.

Page 101: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

101

Command

neighbor {ip-address | peer-group-name} update-source interface-type

To have the Cisco IOS software allow Border Gateway Protocol (BGP) sessions to use a specific operational interface for TCP connections, use the neighbor update-source command in router configuration mode.

To restore the interface assignment to the closest interface, which is called the best local address, use the no form of this command.

Page 102: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

102

Command maximum-paths maximum-number

To control the maximum number of parallel routes an IP routing protocol can support, use the maximum-paths command in address family or router configuration mode. To restore the default value (1), use the no form of this command.

Page 103: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

103

Load Balancing

BGP is not designed to load-balance traffic BGP chooses and installs one “best” route “Attempting” to balance traffic comes in two parts

Inbound trafficOutbound traffic

Load balancing is possible in some topologiesA pair of eBGP peers connected via multiple linksTwo connections from one router to the same AS

But not othersMulti-homed to more than one provider

Page 104: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

104100200

A Loopback 02.2.2.2/32

Single Path Router A: interface loopback 0 ip address 1.1.1.1 255.255.255.255 ! router bgp 100 neighbor 2.2.2.2 remote-as 200 neighbor 2.2.2.2 update-source loopback0 neighbor 2.2.2.2 ebgp-multi-hop ! ip route 2.2.2.2 255.255.255.255 serial 0 ip route 2.2.2.2 255.255.255.255 serial 1 !

B

RA must do a recursive lookup for 2.2.2.2RA has two equal cost paths to 2.2.2.2

RA will load balance traffic over these two links Both inbound and outbound traffic will be balanced

Page 105: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

105

eBGP Multipath Support

Router peers with multiple routers in the same neighbor AS Install multiple routes in IP routing table

Use ‘maximum-paths’ command Routes must be identical in terms of LOCAL_PREF, AS_PATH, MED, etc…

Outbound traffic will be split over these two links BGP still advertises one best path Next-hop is set to self (use loopback interface)

100 200

Page 106: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

106

AS 100

AS 200

AS 300

AA

BB CC

DD

Multi-Homed AS

Very common topology for many customers Customer wants to split traffic between AS 100 and AS 300 Misconception: “I’ll make half of my routes preferred via AS

100 and the other half through AS 300. Then I’ll have load-balancing!!”

Page 107: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

107

Multi-Homed AS This does not provide load balancing, just “prefix splitting” Traffic may be balanced perfectly until traffic patterns

change Load balancing is now over :( Some customers use this method but they are forced to

change their policies to accommodate for changes in traffic patterns

For outbound balancing use Weight LOCAL_PREF (recommended)

For inbound balancing use Conditional-advertisement AS_PATH prepending (may not work) MEDs (may not work) Communities and LOCAL_PREF (recommended)

Page 108: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case 3 : BGP community

community-list route-map

Page 109: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

109

Page 110: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

110

Demand

The backup path of KOREN access TEIN2 backbone via CERNET

Backup link

Page 111: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

111

Demand (Contd..) The backup path of TEIN2

Beijing POP and TEIN2 Singapore POP via CERNET-KOREN connection

Backup link

Page 112: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

112

The detail demands CERNET advertises all the routers from TEIN2 Peking POP (containi

ng community 24489:65200 ) except for the router containing community 24489:65500 to KOREN, advertising such routers to KOREN, CERNET does the as-path-prepending (adding one “4538” ) in order to avoid KOREN selecting the path from CERNET to TEIN2-NORTH priorityr;

CERNET receives all the routers ( containing community 24489:65200 ) transmitting from TEIN2 Singapore POP from KOREN, adds community 4538:9270, and advertises those routers to TEIN2 Peking POP;

CERNET receives all the routers ( containing community 9270:65155 ) transmitting from KOREN internal R&E network from KOREN, adds the community 4538:9270, and advertises those routers to the TEIN2 Peking POP;

When the routers containing the community 4538:9270 are received from TEIN2 Peking POP, and the routers containing the community 4538:9270 are advertised to TEIN2 Peking POP, any restricting and adjusting should not be done, including the IPv4 and IPv6 router.

Page 113: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

113

Configuration

KORENTEIN2

AS9270

.85 .86

AS24489

.25

.26 CERNET

202.179.241/302001:254:1:7::2/64

202.112.61/302001:250:0:30::1/64

AS4538

router bgp 4538 bgp router-id 202.112.60.247 no bgp fast-external-fallover bgp log-neighbor-changes bgp deterministic-med neighbor 202.112.61.86 remote-as 9270 neighbor 202.112.61.86 description KOREN neighbor 202.112.61.86 activate neighbor 202.112.61.86 send-community neighbor 202.112.61.86 soft-reconfiguration inbound neighbor 202.112.61.86 route-map KORENtoTEIN2 in neighbor 202.112.61.86 route-map TEIN2toKOREN out neighbor 202.112.61.86 activate neighbor 202.112.61.86 soft-reconfiguration inbound neighbor 202.112.61.86 filter-list 12 out neighbor 202.179.241.25 remote-as 24489 neighbor 202.179.241.25 description TEIN2 neighbor 202.179.241.25 password 7 02040D550C0A0A774F410615 neighbor 202.179.241.25 activate neighbor 202.179.241.25 send-community neighbor 202.179.241.25 soft-reconfiguration inbound neighbor 202.179.241.25 distribute-list 26 in neighbor 202.179.241.25 route-map tein2-in in neighbor 202.179.241.25 route-map CERNETtoTEIN2 out

Page 114: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

114

Configuration (Contd..)

KORENTEIN2

AS9270

.85 .86

AS24489

.25

.26 CERNET202.179.241/30

2001:254:1:7::2/64202.112.61/30

2001:250:0:30::1/64

AS4538

route-map CERNETtoTEIN2 permit 10 match community CERNETtoTEIN2!route-map CERNETtoTEIN2 permit 20 match ip address prefix-list CERNET2 set community 4538:65155 additive

route-map TEIN2toKOREN permit 10 match community TEIN2bjpop set as-path prepend 4538!route-map TEIN2toKOREN permit 20 match as-path 12 match community TEIN2bjpop1!route-map TEIN2toKOREN permit 30 match ip address prefix-list CERNET2!

route-map KORENtoTEIN2 permit 10 match community KORENtoTEIN2 set local-preference 330 set as-path prepend 4538 set community 4538:9270 additive!route-map KORENtoTEIN2 permit 20 match ip address prefix-list KOREN-filter set local-preference 330 set as-path prepend 4538 set community 4538:9270 additive

route-map tein2-in permit 10 set local-preference 330 set community 4538:24489 additive!

Page 115: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

115

Configuration (Contd..)

KORENTEIN2

AS9270

.85 .86

AS24489

.25

.26 CERNET

202.179.241/302001:254:1:7::2/64

202.112.61/302001:250:0:30::1/64

AS4538

ip community-list standard TEIN2bjpop deny 24489:65500ip community-list standard TEIN2bjpop permit 24489:65200ip community-list standard TEIN2bjpop1 deny 24489:65500ip community-list standard KORENtoTEIN2 permit 24489:65200ip community-list standard KORENtoTEIN2 permit 9270:65155ip community-list standard CERNETtoTEIN2 permit 4538:9270 9270:65155ip community-list standard CERNETtoTEIN2 permit 4538:9270 24489:65200

Page 116: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

116

Show commandsbj-bgw-r0a#sh ip bgp sumBGP router identifier 202.112.60.247, local AS number 4538BGP table version is 61059, main routing table version 61059219479 network entries using 22167379 bytes of memory353680 path entries using 16976640 bytes of memory53580 BGP path attribute entries using 3000592 bytes of memory74 BGP rrinfo entries using 1776 bytes of memory41271 BGP AS-PATH entries using 1169092 bytes of memory2465 BGP community entries using 173974 bytes of memory2 BGP extended community entries using 48 bytes of memory11515 BGP route-map cache entries using 368480 bytes of memory776 BGP filter-list cache entries using 9312 bytes of memoryBGP using 43867293 total bytes of memory249339 received paths for inbound soft reconfigurationBGP activity 228135/1064 prefixes, 768702/399806 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd193.62.157.29 4 786 0 0 0 0 0 never Idle202.112.60.246 4 4538 576 161 61059 0 0 02:37:48 16791202.112.60.251 4 4538 4252 4407 61059 0 0 02:37:53 47349202.112.61.14 4 23911 89332 192 61059 0 0 02:38:39 4202.112.61.46 4 9264 775 344 61059 0 0 02:37:40 132202.112.61.86 4 9270 2294 913 61055 0 0 02:38:50 10038202.112.61.94 4 9505 215 186 61059 0 0 02:37:41 366202.179.241.25 4 24489 3896 420 61055 0 0 02:38:44 9589203.181.194.125 4 7660 4626 1275 61055 0 0 02:38:19 10036203.181.194.126 4 7660 9799 1116 61055 0 0 02:37:16 10031203.222.180.225 4 1239 44795 215 61059 0 0 02:37:30 3bj-bgw-r0a#

Page 117: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Bgp session end

Page 118: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Trouble Shooting OSPF

Page 119: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

119

Agenda

LSA Overview Troubleshooting Commands Common Issues

Page 120: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

120

LSA OVERVIEWLSA OVERVIEW

© 2004 Cisco Systems, Inc. All rights reserved.

RST-33019721_05_2004_c2 120120120

Page 121: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

121

LSA Type Review

Type LSA

1 Router

2 Network

3 Summary Network

4 Summary ASBR

5 External

6 Group Membership

7

9–11

NSSA

External Attributes

Opaque

8

Page 122: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

122

Router LSA Details Router LSA (Type 1)

Describes the state and cost of the router’s links to the area

All of the router’s links in an area must be described in a single LSA

Flooded throughout the particular area and no more

Router indicates whether it is an ASBR, ABR, or end point of virtual link

Page 123: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

123

192.1.4.0/24

192.1.1.4

192.1.1.3

Area 0

R3#show ip ospf database router 3.3.3.3Router Link States (Area 1)

LS age = 0 Always 0 at originationOptions = (E-bit)LS type = 1 This is a router LSALink State ID = 3.3.3.3 Router ID of R3Advertising Router = 3.3.3.3 Router ID of R3It is an area border router bit B = 1 # links = 2

Link ID = 192.1.1.4 IP address of the DRLink Data = 192.1.1.3 Interface address of this router

Type = 2 This is a transit network # TOS metrics = 0

metric = 1 Cost to reach the interfaceLink ID = 192.1.4.0 IP network numberLink Data = 255.255.255.0 Subnet mask of the interfaceType = 3 Stub network# TOS metrics = 0metric = 2

DR

R4

2R3

1

Router LSA of R3 for Area 1

Page 124: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

124

Router Link States (Area 0)

LS age = 0Options = (E-bit)LS type = 1Link State ID = 3.3.3.3Advertising Router = 3.3.3.3It is an area border router bit B = 1 # links = 2

Link ID = 6.6.6.6 Router id of the neighborLink Data = 18.10.0.5 IP interface address of the router Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8Link ID = 18.10.0.4 IP subnet address

Link Data = 255.255.255.252 Subnet mask Type = 3 This is a stub link # TOS metrics = 0 metric = 8

Area 0

DR

1

6.6.6.6

Router LSA of R3 for Area 0 (Cont.)

R6

192.1.1.3

R3

R4192.1.1.4

192.1.4.0/24

8

18.10.0.5/30

2

Page 125: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

125

Router LSA DetailsType Description Link ID Link Data

11Point-to-PointPoint-to-Point

NumberedNumberedNeighbors’Neighbors’

RIDRIDInterface IPInterface IP

AddressAddress

11Point-to-PointPoint-to-PointUnnumberedUnnumbered

Neighbors’Neighbors’RIDRID

MIB-II MIB-II IfindexIfindexValueValue

22 TransitTransitIP AddressIP Addressof the DRof the DR

Interface IPInterface IPAddressAddress

33 StubStub IP NetworkIP NetworkNumberNumber Subnet MaskSubnet Mask

44 Virtual LinkVirtual Link Neighbors’Neighbors’RIDRID

Interface IPInterface IPAddressAddress

Page 126: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

126

Network LSA

Network LSA (Type 2) Generated for every transit broadcast

and NBMA network Describes all the routers attached to

the network Only the designated router originates

this LSA Flooded throughout the area and no

more

Page 127: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

127

R3#show ip ospf database network 192.1.1.4Network Link States (Area 1)

LS age = 948Options = (E-bit)LS type = 2Link State ID = 192.1.1.4 IP interface address of DRAdvertising Router = 4.4.4.4 RID of DRNetwork Mask = 255.255.255.0

Attached Router = 4.4.4.4Attached Router = 3.3.3.3 RID of attached routers FULL with the DRAttached Router = 2.2.2.2Attached Router = 1.1.1.1

Network LSA for 192.1.1.0

192.1.1.3

DR

R4

192.1.4.0/24

18.10.0.4/32

6.6.6.6

R3

192.1.1.4

1.1.1.1

2.2.2.2

R1

R2

/241

2

8

R6

4.4.4.4

Area 0

Page 128: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

128

Summary LSA Describes the destination outside the area

but still in the AS Summary is created for each IP subnets in

one area and is flooded out in all other areas

Originated by an ABR Only intra-area routes are advertised into

the backbone Type 4 is the information about the ASBR

Page 129: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

129

R4#show ip ospf database summary 192.1.2.0Summary Net Link States (Area 0)

LS age = 1514Options = (E-bit)LS type = 3Link State ID = 192.1.2.0 IP network numberAdvertising Router = 4.4.4.4 RID of ABRNetwork Mask = 255.255.255.0metric = 4

Type 3 Details

192.1.1.3

Area 0

192.1.4.0/24

18.10.0.4

6.6.6.6

R3

192.1.1.4

1.1.1.1

2.2.2.2

R2

/24

1

2

8

8192.1.2.0/24

ABR 192.1.2.0/24 metric 4

O IA

DR

R4R1

3

R6

ABR

Page 130: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

130

R4#show ip ospf database asbr-summary 7.7.7.7Summary ASB Link States (Area 0)

LS age = 1548Options = (E-bit)LS type = 4Link State ID = 7.7.7.7 RID of ASBRAdvertising Router = 4.4.4.4 RID of ABRNetwork Mask = 0.0.0.0Metric = 16

Type 4 Details

18.10.0.4

R3

1.1.1.1

2.2.2.2

/24 Area 01

8

192.1.4.0/24

R2

8

192.1.2.0/24

DR

R4R1

3

140.10.0.0

External Route

R7

RID ASBR 7.7.7.7

8

R3

ABRType 4 Summary

2

6.6.6.6

R6

192.1.1.3

192.1.1.4

(4.4.4.4)

Page 131: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

131

External LSA

External LSA (Type 5) Defines routes to destination external

to the AS Default route is also sent as external Two types of external LSA:

E1: Consider the total cost up to the external destination

E2: Considers only the cost of the outgoing interface to the external destination

Page 132: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

132

R4#show ip ospf database external 140.10.0.0LS age = 1156Options = (E-bit)LS type = 5Link State ID = 140.10.0.0 IP network numberAdvertising Router = 7.7.7.7 Router ID of R7Network Mask = 255.255.0.0Metric Type: 2 bit E = 1 -> O E2 (Default)metric = 20 The metric is 20 in all redistributed E2 routesForwarding address = 0.0.0.0 Traffic should be forwarded to the ASBR

1.1.1.1

Area 0

192.1.4.0/24R2

192.1.2.0/24

DR

R4R1

R7

RID ASBR 7.7.7.7

8

R3 2 18.10.0.4

6.6.6.6192.1.1.3

192.1.1.4

2.2.2.2

/24

1

8

3140.10.0.0

External Route

External Type 5R6

Type 5 Details

Page 133: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

133

NSSA External LSA NSSA External LSA (Type 7) RFC1587

NSSA was created to inject external routes from stub area into OSPF domain

Redistribution in NSSA creates Type 7 LSA Generated by the NSSA ASBR Type 7 can only exist in NSSA area NSSA ABR does the translation from 7–5

Page 134: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

134

TROUBLESHOOTING COMMANDSTROUBLESHOOTING COMMANDS

© 2004 Cisco Systems, Inc. All rights reserved.

RST-33019721_05_2004_c2 134134134

Page 135: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

135

R3#show ip ospf Routing Process "ospf 1" with ID 3.3.3.3 and Domain ID 0.0.0.1 Supports only single TOS(TOS0) routes Supports opaque LSA It is an area border router SPF schedule delay 5 secs, Hold time between two SPFs 10 secs Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs Number of external LSA 1. Checksum Sum 0x3B57 Number of opaque AS LSA 0. Checksum Sum 0x0 Number of DCbitless external and opaque AS LSA 0 Number of DoNotAge external and opaque AS LSA 0 Number of areas in this router is 2. 2 normal 0 stub 0 nssa External flood list length 0 Area BACKBONE(0) Number of interfaces in this area is 2 Area has no authentication SPF algorithm executed 2773 times Area ranges are Number of LSA 17. Checksum Sum 0x686B5 Number of opaque link LSA 0. Checksum Sum 0x0 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 9 Flood list length 0

Show IP OSPF

Page 136: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

136

. . .Area 1 Number of interfaces in this area is 2 Area has no authentication SPF algorithm executed 22 times Area ranges are Number of LSA 19. Checksum Sum 0x8FE73 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 0 Flood list length 0

Show IP OSPF (Cont.)

Page 137: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

137

R3#show ip ospf database OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 0) Link ID ADV Router Age Seq# Checksum Link count3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3 . . . Summary Net Link States (Area 0)

Link ID ADV Router Age Seq# Checksum18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2 18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8 . . . Router Link States (Area 1)

Link ID ADV Router Age Seq# Checksum Link count1.1.1.1 1.1.1.1 671 0x80000016 0xE6CD 2. . .

Show IP OSPF Database

Page 138: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

138

R3#show ip ospf database database-summary

OSPF Router with ID (3.3.3.3) (Process ID 1)

Area 0 database summary LSA Type Count Delete Maxage Router 6 0 0 Network 4 0 0 Summary Net 10 0 0 Summary ASBR 0 0 0 Type-7 Ext 0 0 0 Opaque Link 0 0 0 Opaque Area 0 0 0 Subtotal 20 0 0

Area 1 database summary LSA Type Count Delete Maxage Router 4 0 0 Network 1 0 0 Summary Net 10 0 0 Summary ASBR 4 0 0 . . .

Show IP OSPF Database Database-Summary

Page 139: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

139

R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface1.1.1.1 1 FULL/DROTHER 00:00:33 192.1.1.1 FastEthernet0/02.2.2.2 1 FULL/DROTHER 00:00:32 192.1.1.2 FastEthernet0/04.4.4.4 1 FULL/DR 00:00:39 192.1.1.4 FastEthernet0/06.6.6.6 1 FULL/ - 00:00:38 18.10.0.6 Serial0/0R3#

Show IP OSPF Neighbor

192.1.1.3

DR

R4

192.1.4.0/24

18.10.0.4

6.6.6.6

R3

192.1.1.4

1.1.1.1

2.2.2.2

R1

R2

/241

2

8

R6

4.4.4.4

Area 0

Page 140: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

140

R3#config terminalR3(config)#router ospf 1R3(config-router)#log-adjacency-changes%OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on FastEthernet0/0 from 2WAY to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from FULL to DOWN, Neighbor Down: Interface down or detached%LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from LOADING to FULL, Loading Done%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from LOADING to FULL, Loading Done

OSPF Log-Adjacency-Changes Default as of 12.1.3 and 12.0.12S

Page 141: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

141

R3#show ip ospf neighbor detailNeighbor 1.1.1.1, interface address 192.1.1.1 In the area 1 via interface FastEthernet0/0 Neighbor priority is 1, State is 2WAY, 2 state changes DR is 192.1.1.4 BDR is 192.1.1.2 Options is 0x2 Dead timer due in 00:00:39 Neighbor is up for 00:06:30 Index 0/0, retransmission queue length 0, number of retransmission 0 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 0, maximum is 0 Last retransmission scan time is 0 msec, maximum is 0 msec Neighbor 2.2.2.2, interface address 192.1.1.2 In the area 1 via interface FastEthernet0/0 Neighbor priority is 1, State is FULL, 6 state changes DR is 192.1.1.4 BDR is 192.1.1.2 Options is 0x42 Dead timer due in 00:00:38 Neighbor is up for 00:06:31 Index 2/2, retransmission queue length 0, number of retransmission 0 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 0, maximum is 0 Last retransmission scan time is 0 msec, maximum is 0 msec

Show IP OSPF Neighbor Detail

Page 142: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

142

R3#show ip ospf interfaceFastEthernet0/0 is up, line protocol is up Internet Address 192.1.1.3/24, Area 1 Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 1 Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 4.4.4.4, Interface address 192.1.1.4 Backup Designated router (ID) 2.2.2.2, Interface address 192.1.1.2 Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:03 Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 5 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 3, Adjacent neighbor count is 2 Adjacent with neighbor 2.2.2.2 (Backup Designated Router) Adjacent with neighbor 4.4.4.4 (Designated Router) Suppress hello for 0 neighbor(s)

Show IP OSPF Interface

Page 143: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

143

R3#show ip ospf virtual-linksVirtual Link OSPF_VL0 to router 4.4.4.4 is up Run as demand circuit DoNotAge LSA allowed. Transit area 1, via interface FastEthernet0/0, Cost of using 1 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:09 Adjacency State FULL (Hello suppressed) Index 1/3, retransmission queue length 0, number of retransmission 1 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 1, maximum is 1 Last retransmission scan time is 0 msec, maximum is 0 msecR3#

Show IP OSPF Virtual-Links

Page 144: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

144

R3#sh ip ospf stat Area 0: SPF algorithm executed 42 times Area 1: SPF algorithm executed 38 times

SPF calculation timeDelta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason00:22:00 0 0 0 0 0 0 0 R, N, SN, 00:21:44 0 0 4 0 0 0 4 R, SN, X00:21:34 0 0 4 0 0 0 4 R, SN, X00:21:24 0 0 0 4 0 0 4 R, SN, X00:21:14 0 0 0 0 0 0 0 R, 00:21:04 0 0 0 0 0 0 0 R, N, SN, 00:20:54 0 0 0 0 0 0 0 X00:20:44 0 0 4 0 0 0 4 R, SN, X00:20:34 0 0 0 0 0 0 0 X00:00:17 4 0 0 0 0 0 4 R, N, SN, SA, X. . .R=Router LSA; N=NetworkLSA; SN=Summary Network LSA; SA=Summary ASBR LSA; X=External LSA

Show IP OSPF Stat Requires enable mode

Page 145: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

145

R3#show ip ospf borders-routersOSPF Process 1 internal Routing Table

Codes: i - Intra-area route, I - Inter-area route

i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 0, SPF 42i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 1, SPF 38i 8.8.8.8 [10] via 18.10.0.6, Serial0/0, ABR/ASBR, Area 0, SPF 42i 7.7.7.7 [17] via 192.1.1.4, FastEthernet0/0, ABR/ASBR, Area 0, SPF 42

Show IP OSPF Borders

Page 146: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

146

Other Show CommandsR3#show ip ospf database self-originate OSPF Router with ID (3.3.3.3) (Process ID 1)

Router Link States (Area 0)Link ID ADV Router Age Seq# Checksum Link

count3.3.3.3 3.3.3.3 1520 0x80000015 0xABFD 2 Summary Net Link States (Area 0)Link ID ADV Router Age Seq# Checksum192.1.1.0 3.3.3.3 1520 0x80000006 0x4E1A 192.1.2.0 3.3.3.3 1521 0x80000006 0x6103 . . . Router Link States (Area 1)Link ID ADV Router Age Seq# Checksum Link

count3.3.3.3 3.3.3.3 1536 0x80000028 0x612D 2

Page 147: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

147

Other Show Commands (Cont.)

R3#show ip ospf database adv-router 7.7.7.7OSPF Router with ID (3.3.3.3) (Process ID 1) Router Link States (Area 0)Link ID ADV Router Age Seq# Checksum Link

count7.7.7.7 7.7.7.7 871(DNA) 0x8000000D 0x8FE2 2 Summary Net Link States (Area 0)Link ID ADV Router Age Seq# Checksum20.10.0.0 7.7.7.7 871 (DNA) 0x8000000A 0x39C4 Type-5 AS External Link StatesLink ID ADV Router Age Seq# Checksum Tag140.100.0.0 7.7.7.7 1944 0x80000004 0x3759 0

Page 148: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

148

COMMON ISSUESCOMMON ISSUES

148148148

Page 149: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

149

Common Issues

Adjacency is not coming up OSPF neighbor stuck in ? state Information is in the DB but not in the

RT SPF running constantly Neighbor flapping (Frame Relay) NSSA ABR not translating Type 7 LSA Demand circuit problems

Page 150: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

150

Adjacency Is Not Coming Up

Useful commands for this problem Show IP OSPF neighbor Show IP OSPF interface Debug IP OSPF adjacency

Page 151: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

151

R3#show ip ospf neighborR3#

R3#show ip ospf interface serial 2Serial2 is down, line protocol is DOWN Internet Address 18.10.0.3/30, Area 0 Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64 Transmit Delay is 1 sec, State DOWN, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5

Layer 2 is down

Adjacency Is Not Coming Up

Page 152: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

152

R3#show ip ospf neighborR3#

R3#show ip ospf interface serial 2Serial2 is up, line protocol is up OSPF not enabled on this interface

In 12.0:

R3#show ip ospf interface serial 2R3#

Tip: Check for the wrong network statement re-enter the network statement

Adjacency Is Not Coming Up

OSPF not enabled on the interface

Page 153: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

153

R3#show ip ospf neighborR3#

R3#show ip ospf interface e0Ethernet0 is up, line protocol is up Internet Address 192.1.1.3/24, Area 1 Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 192.1.1.4, Interface address 192.1.1.3 No backup designated router on this network Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 No Hellos (Passive interface)

Adjacency Is Not Coming Up

Interface is defined as passive

Page 154: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

154

Adjacency Is Not Coming Up

Mismatched subnet mask

R3#show ip ospf neighborR3#

R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Mismatched hello parameters from 192.1.1.4Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.192 C 255.255.255.0

Page 155: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

155

R3#show ip ospf neighborR3#

R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Mismatched hello parameters from 192.1.1.4Dead R 40 C 40, Hello R 15 C 10 Mask R 255.255.255.0 C 255.255.255.0

R4(config-if)#interface ethernet 0R4(config-if)#no ip ospf hello-interval 15

Tip: Default is 10 second on LAN

Adjacency Is Not Coming Up Mismatched hello/dead interval

Page 156: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

156

R3#show ip ospf neighborR3#

R3#debug ip ospf adjOSPF adjacency events debugging is onR3#OSPF: Rcv pkt from 192.1.1.4, Ethernet0 : Mismatch Authentication Key - Clear Text

Tip: Watch for the “space” at the end of the Authentication key

Adjacency Is Not Coming Up

Mismatched authentication key

Page 157: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

157

OSPF: Rcv pkt from 192.1.1.4, Ethernet0, area 0.0.0.1 mismatch area 0.0.0.2 in the header

Neighbor is in area 2 but we are not:

%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from backbone area must be virtual-link but not found from 192.1.1.4, Ethernet0

R4#show ip ospf neighborR4#

R4#debug ip ospf adjOSPF adjacency events debugging is on

Adjacency Is Not Coming Up

Mismatched area ID

Page 158: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

158

OSPF: Hello from 18.10.0.2 with mismatched Stub/Transit area option bit

Adjacency Is Not Coming Up

Mismatched transit/stub/NSSA option

R7#show ip ospf neighborR7#

R7#debug ip ospf adjOSPF adjacency events debugging is on

7.7.7.7 8.8.8.8

R8R7area 2

area 2 nssa

18.10.0.218.10.0.1

Page 159: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

159

OptionsNormal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372E bit is 0, no external allowed, options: 0x0 = 00000000NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372N/P bit is on, options: 0x8 = 00001000DC: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392DC bit is negotiated, options: 0x20 = 00100000

* *DC EA N/P MC EO

Page 160: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

160

OSPF Neighbor Stuck in ? State

Useful commands for this problem Show IP OSPF neighbor Debug IP OSPF adjacency

Page 161: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

161

RID =6.6.6.6

N = 3.3.3.3

Hello

Stuck in ATTEMPT

3.3.3.3 6.6.6.6

RID =3.3.3.3

Hello

R3 R6

NBMA

Page 162: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

162

Stuck in ATTEMPT

Reasons: Our hellos are getting lost in NBMA cloud Neighbor hellos are getting lost in NBMA

cloud We received neighbor’s hello but rejects

it for some reason Misconfigured neighbor statement Broken Unicast

Page 163: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

163

RID =2.2.2.2

Hello

Stuck in INIT 2.2.2.2

RID =1.1.1.1

Hello

R2

RID =1.1.1.1

Hello

N =2.2.2.2

RID =2.2.2.2

Hello

1.1.1.1

R1

Page 164: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

164

Stuck in INIT

Reasons: One side is blocking the hello packet

with access-list One side is translating (NAT) ospf hello One side multicast capabilities is broken

(Layer 2) Dialer map or frame-relay map is missing

keyword ‘broadcast’

Page 165: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

165

Stuck in 2-WAY 1.1.1.1 2.2.2.2

RID =1.1.1.1 P=0

Hello

RID =1.1.1.1 P=0

Hello

N =2.2.2.2

R2R1

RID =2.2.2.2 P=0

Hello

N =1.1.1.1

Page 166: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

166

Stuck in 2-WAY Reasons:

This is normal in broadcast network types This is to reduce the amount of flooding on the

wire Problem can happen if all the router are

configured with priority equal to ‘0’

In a situation where you have high and low end boxes on the same segment the configure low end routers with priority 0 so they don’t participate in DR election

Page 167: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

167

Stuck in EXSTART/EXCHANGE

3.3.3.3 6.6.6.6

RID =3.3.3.3

HelloR6R3

RID =6.6.6.6

Hello

N =3.3.3.3

MTU = 1500 flag = 0x7

DBD

Seq = 1E55

MTU = 1500 flag = 0x7

DBD

Seq = 22AB

Page 168: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

168

Stuck in EXSTART/EXCHANGE Useful in debugging, defines I, M and MS bits

OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492

Flag 0x7--> 111 means I(Initial) = 1, M = 1(More), MS = 1(Master) Flag 0x6 --> 110 not possible Flag 0x5 --> 101 not possible Flag 0x4 --> 100 not possible Flag 0x3 --> 011 means master has more data to send Flag 0x2 --> 010 means slave has more data to send Flag 0x1 --> 001 means master has no more data left to send Flag 0x0 --> 000 means slave has no more data left to send

0 00 0 I

M MS0

Page 169: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

169

Stuck in EXSTART/EXCHANGE Reasons:

MTU mismatch Note: If Cisco IOS is < 12.0.3 neighbor will show stuck in EXCHANGE

Neighbor RID is same as ours.Note: If Cisco IOS is > 12.0.7, it displays msg: %OSPF-3-DUP_RTRID & OSPF neighbor list will be empty

Unicast is brokena. Wrong VC/DLCi mapping in frame/ATM environment in highly redundant networkb. MTU problem, can’t ping across with more than certain length packet c. Access-list blocking unicast; after 2-way OSPF send unicast packet except p2p linksd. NAT is translating unicast packet

Between PRI and BRI/dialer and network type is p2p

Page 170: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

170

Stuck in LOADING3.3.3.3 6.6.6.6

R6R3

LS Type

LS Req

Link State ID

Advertising Router

# LSAs

LS Update

LSAs. . .

Page 171: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

171

Stuck in LOADING Reasons:

LS request is being made and neighbor is sending bad packet or mem corrupt

a. Do show IP OSPF bad to see bad lsa b. Show log will show

OSPF-4-BADLSATYPE msg LS request is being made and neighbor is

ignoring the request

MTU mismatch problem (RFC 1583 and 2178 compatibility issue)

Page 172: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

172

Stuck in LOADING3.3.3.3 6.6.6.6

R6R3

LS Type

LS Req

Link State ID

Advertising Router

# LSAs

LS Update

LSAs

. . .

MTU = 2048

IOS 11.3.10T

RFC 1583

MTU = 4470

IOS 12.0.7T

RFC 2178

Too Big!

Page 173: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

173

Information Is in the DBbut Not in the RT

Useful commands for this problem Show IP OSPF interface <interface> Show IP OSPF database <x> Where ‘x’ can be router, network,

summary, summary-asbr, external, nssa

Page 174: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

174

R3#show ip ospf interface serial 0Serial0 is up, line protocol is up Internet Address 18.0.0.5/30, Area 0 Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64

R6#show ip ospf interface serial 0Serial0 is up, line protocol is up Internet Address 18.0.0.6/30, Area 0 Process ID 1, Router ID 6.6.6.6, Network Type BROADCAST, Cost: 64

Mismatched Network Types

Area 0

R3

3.3.3.3

R6

6.6.6.6

18.10.0.4/30

R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0

Page 175: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

175

R618.10.0.4/30

Area 0

R3

3.3.3.3 6.6.6.6

R3#show ip ospf database router 3.3.3.3. . .

Link ID = 6.6.6.6 Router id of the neighborLink Data = 18.10.0.5 IP interface address Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8

. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 18.10.0.6 IP address of the DR Link Data = 18.10.0.6 Interface address Type = 2 This is a transit link # TOS metrics = 0 metric = 8

Mismatched Network Types (Cont.)

Page 176: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

176

Point-to-Point Numbered and Unnumbered Links

R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0

R3#show interface serial0 Serial0 is up, line protocol is up Hardware is HD64570 Interface is unnumbered.Using address of Ethernet1 (192.1.4.1)

R6#show interface serial0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.6/30

Area 0

R3

3.3.3.3

R6

6.6.6.6

18.10.0.4/30

192.1.4.1

Page 177: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

177

R618.10.0.4/30

Area 0

R3

3.3.3.36.6.6.6

R3#show ip ospf database router 3.3.3.3. . .

Link ID = 6.6.6.6 Router id of the neighborLink Data = 0.0.0.5 MIBII IfIndex Value Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8

. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 3.3.3.3 Router id of the neighbor Link Data = 18.10.0.6 IP interface address Type = 1 This is a transit link # TOS metrics = 0 metric = 8

192.1.4.1

Point-to-Point Numbered and Unnumbered Links (Cont.)

Page 178: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

178

R3#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 19.10.0.5/24

R6#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.6/30

Different Mask or IP Subneton p2p Links

Area 0

R3

3.3.3.3

R6

6.6.6.6

19.10.0.5/30

R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.0.0.6 Serial0

18.10.0.6/30

Page 179: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

179

R3#show ip ospf database router 3.3.3.3. . .

Link ID = 6.6.6.6 Router id of the neighborLink Data = 19.10.0.5 Interface address Type = 1 This is a point-to-point link # TOS metrics = 0metric = 8

. . .R3#show ip ospf database router 6.6.6.6. . . Link ID = 3.3.3.3 Router id of the neighbor Link Data = 18.10.0.6 Interface address Type = 1 This is a point-to-point link # TOS metrics = 0 metric = 8

R6

Area 0

R3

3.3.3.3 6.6.6.6

Different Mask or IP Subneton p2p Links (Cont.)

Page 180: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

180

Address Flipped on Dual Links

R3#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 6.6.6.6 1 FULL/ - 00:00:30 18.10.0.6 Serial0 6.6.6.6 1 FULL/ - 00:00:33 18.10.0.22 Serial1

R3#show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 Internet address is 18.10.0.21/30

R3

3.3.3.3

R6

6.6.6.6

18.10.0.20/30

Area 0

18.10.0.4/30S0

S1

Page 181: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

181

Forwarding Address Problem

R1#show ip ospf database external 140.10.0.0. . .Link State ID = 140.10.0.0Advertising Router = 5.5.5.5Network Mask = 255.255.0.0. . .Forwarding address = 18.10.0.10

R1#show ip route 18.10.0.10Routing entry for 18.10.0.8/29 Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 10. . .. . .

R5R4 R7R1

18.10.0.1/30

18.10.0.9/29

18.10.0.5/30RIP

Area 1Area 0

140.10.0.0

External Route

Page 182: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

182

Forwarding Address Problem

R5:

router ospf 1 network 18.10.0.0 0.0.0.255 area 1 redistribute rip subnets redistribute connected subnets ! router rip network 10.0.0.0

R4:

router ospf 1 area 1 range 18.10.0.0 255.255.255.240

R5 R7

18.10.0.1/30

18.10.0.5/30Area 1

Area 0140.10.0.0

External Route

R4R1 18.10.0.9/29

RIP

Page 183: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

183

ABR2ABR1 R2R1Area 1

Area 0 Area 0

Discontigous Backbone

R1 and R2 are not be able to see each other Summary LSA for Inter-area routes must not

be generated into the backbone The solution is to create virtual link between

ABR1 and ABR2

Summary LSA

Summary LSA

Summary LSA

Summary LSA

Area 0

Page 184: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

184

Distribute-List in Blocking the Routes

R4#show ip route 18.10.0.9 % Subnet not in table

R4:

router ospf 1 network 18.10.0.0 0.0.0.255 area 1 distribute-list 1 in ! access-list 1 permit 18.10.0.0 0.0.0.3

R5R4

R7

18.10.0.1/30

18.10.0.9/29

Page 185: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

185

SPF Running Constantly

Useful commands for this problem Show IP OSPF stat Show IP OSPF database Show IP OSPF database database-sum

Page 186: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

186

SPF Running Constantly

Reasons: LSA flaps due to:

Duplicate RID/IP address Constant link flapping in an area

Page 187: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

187

R3#sh ip ospf stat Area 0: SPF algorithm executed 42 times Area 1: SPF algorithm executed 38 times

SPF calculation timeDelta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason00:22:00 0 0 0 0 0 0 0 R, N, SN, 00:21:44 0 0 4 0 0 0 4 R, SN, X00:21:34 0 0 4 0 0 0 4 R, SN, X00:21:24 0 0 0 4 0 0 4 R, SN, X00:21:14 0 0 0 0 0 0 0 R, 00:21:04 0 0 0 0 0 0 0 R, N, SN, 00:20:54 0 0 0 0 0 0 0 X00:20:44 0 0 4 0 0 0 4 R, SN, X00:20:34 0 0 0 0 0 0 0 X00:00:17 4 0 0 0 0 0 4 R, N, SN, SA, X. . .R=Router LSA; N=NetworkLSA; SN=Summary Network LSA; SA=Summary ASBR LSA; X=External LSA

SPF Running Constantly Requires enable mode

Page 188: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

188

R3#deb ip ospf mon

OSPF: schedule SPF in area 1

Change in LS ID 1.1.1.1, LSA type R,

OSPF: schedule SPF: spf_time 0ms wait_interval 861421816s

OSPF: begin SPF at 0x33585480ms, process time 752ms

Spf_time 0ms, wait_interval 861421816s

OSPF: end SPF at 0x33585488ms, total elapsed time 8ms

Intra: 4ms, inter: 0ms, external: 0ms

SPF Running Constantly

Page 189: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

189

R3#show ip ospf database

OSPF Router with ID (3.3.3.3) (Process ID 1)

Router Link States (Area 0)

Link ID ADV Router Age Seq# Checksum Link count

3.3.3.3 3.3.3.3 106 0x80000009 0xC3F1 3

. . .

Summary Net Link States (Area 0)

Link ID ADV Router Age Seq# Checksum

18.10.0.0 7.7.7.7 3 (DNA) 0x80000008 0x3DC2

18.10.0.0 8.8.8.8 1396 0x80000004 0x27D8

. . .

Router Link States (Area 1)

Link ID ADV Router Age Seq# Checksum Link count

1.1.1.1 1.1.1.1 2 0x80000016 0xE6CD 2

. . .

SPF Running Constantly

Page 190: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

190

R3#show ip ospf database database-summary

OSPF Router with ID (3.3.3.3) (Process ID 1)

Area 0 database summary LSA Type Count Delete Maxage Router 124 0 0 Network 4 0 0 Summary Net 10 0 0 Summary ASBR 0 0 0 Type-7 Ext 0 0 0 Opaque Link 0 0 0 Opaque Area 0 0 0 Subtotal 138 0 0

Area 1 database summary LSA Type Count Delete Maxage Router 4 0 0 Network 1 0 0 Summary Net 10 0 0 Summary ASBR 4 0 0 . . .

SPF Running Constantly

Page 191: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

191

Neighbor Flapping (Frame Relay)

Useful commands for this problem Debug ip ospf adj OSPF log-adjacency-change Show IP OSPF neighbors detail Show interface

Page 192: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

192

Area 0

router ospf 1network 10.10.10.10 0.0.0.0 area 0area 0

RID: 7.7.7.7

Type 7–5 Conversion

NSSA ABR Not Translating Type 7 LSA

Only NSSA ABR with the highest RID does the conversion

No Type 7/5 Translation

Area 1NSSA

Type 7

RID: 8.8.8.8

NSSAABR

NSSAASBR

Page 193: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

193

NSSA ABR Not Translating Type 7 LSA

Only NSSA ABR with the highest RID does the conversion

Area 0NSSAABRrouter ospf 1

network 10.10.10.10 0.0.0.0 area 0area 0

RID: 8.8.8.8

NSSAASBR

Type 7

Type 7–5 Conversion

RID: 7.7.7.7

Area 1NSSA

Page 194: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

194

Demand Circuit Problems

The DC is bringing up the link: There is a change in OSPF topology debug IP OSPF monitor is helpful in this case Network type on DC is defined broadcast There is a router in the network that is incapable

to understand DC bit The DC is configured over async interface

(need to configure a dialer interface as a solution)

OSPF (DC)

18.10.0.1/30 18.10.1.0/24

Page 195: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

195

Demand Circuit Problems

DC is bringing up the link (cont.) PPP host route is also owned by RIP, when PPP

host route disappears, the database is change Solution 1: no peer neighbor-route Solution 2: Block /32 route getting into OSPF with

route-map Solution 3: Use different majornet for RIP

RIP—OSPF Redistribution

OSPF (DC)

18.10.0.1/30 18.10.1.0/24

RIP

Page 196: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

196

Summary

What we learned? Overview of OSPF LSAs Different troubleshooting commands

and what to look for in those commands while troubleshooting?

Common issues in OSPF networks; e.g adjacency problems, CPU hogs and SPF problems, NSSA and DC problems etc and how to correct those problems

Page 197: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

© 2002, Cisco Systems, Inc. All rights reserved. 197

OSPF case studies

Page 198: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case 1

Why Does the show ip ospf neighbor Command Reveal Neighbors Stuck in Two-Way

State?

Page 199: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

199

Introduction

This case explains why the show ip ospf neighbor command shows neighbors stuck in a two-way state. It also provides configuration tips.

Page 200: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

200

How OSPF Forms Its Neighbors

In this topology, all routers are running Open Shortest Path First (OSPF) over the Ethernet network

Page 201: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

201

The sample output of the show ip ospf neighbor command

R7# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.3.4 1 2WAY/DROTHER 00:00:34 170.170.3.4 Ethernet0 170.170.

3.3 1 2WAY/DROTHER 00:00:34 170.170.3.3 Ethernet0 170.170.3.8 1 FULL/DR 00:00:32 170.170.3.8 Ethernet0 170.170.3.2 1 FULL/BDR 00:00:39 170.170.3.2 Ethernet0 R8# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.3.4 1 FULL/DROTHER 00:00:37 170.170.3.4 Ethernet0 170.170.3.3 1 FULL/DROTHER 00:00:37 170.170.3.3 Ethernet0 170.170.3.7 1 FULL/DROTHER 00:00:38 170.170.3.7 Ethernet0 170.170.3.2 1 FULL/BDR 00:00:32 170.170.3.2 Ethernet0

Page 202: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

202

• Whenever a router sees itself in a neighbor hello packet, it confirms bidirectional communication and transitions the neighbor state to two-way. At this point, the routers perform DR and BDR election. Once DR and BDR are elected, a router attempts to form a full adjacency with a neighbor if one of the two routers is the DR or BDR. OSPF routers become fully adjacent with routers with which they have successfully completed the database synchronization process. This is the process by which OSPF routers exchange link-state information to populate their databases with the same information. Again, this database synchronization process is only executed between two routers if one of the two routers is the DR or BDR.

Page 203: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

203

Why Do Routers Only Form Full Adjacencies with the DR or BDR?

Page 204: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

204

Why Do Routers Only Form Full Adjacencies with the DR or BDR? (cont.)

Sometimes it is desirable for a router to be configured so that it is not eligible to become the DR or BDR. You can do this by setting the OSPF priority to zero with the ip ospf priority priority# interface subcommand. If two OSPF neighbors both have their OSPF interface priority set to zero, they establish two-way adjacency instead of full adjacency.

Page 205: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

205

example.(topology)

Page 206: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

206

example. The topology below provides an example.

There are three routers connected via Frame Relay. The Frame Relay interfaces are defined as broadcast, but only the router with a connection back to the main network is eligible to be the DR. The other two routers have their interface priorities set to zero, so they are not eligible to become the DR or BDR. Although they do become neighbors, they only reach two-way state.

Page 207: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

207

example. The neighbor table for this topology looks

like this: DRother1# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 170.170.9.5 1 FULL/DR 00:00:30 170.170.9.5 Serial0.5 170.170.10.8 0 2WAY/DROTHER 00:00:38 170.170.9.8 Serial0.5 DRother1#

Notice that, in the figure above, the DRother1 router establishes a two-way adjacency with the DRother2 router.

Page 208: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case 2

OSPF routers do not form neighbor relationships due to authentication type

mismatch

Page 209: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

209

OSPF Errors, Warnings, and Log Messages

Receiving "Mismatch Authentication type."

Page 210: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

210

Core Issue Before exchanging routing information, routers

running Open Shortest Path First (OSPF) form neighbor relationships with other OSPF routers on the same segment. This is done by exchanging hello packets. The hello packets contain various parameters, one of which is related to authentication. This specifies the authentication type and authentication information for the originating interface. Authentication is useful for preventing malicious or incorrect routing information from getting introduced into the routing table. OSPF supports two types of authentication: plain text and Message Digest 5 (MD5).

Page 211: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

211

To resolve this issue, perform these steps:

1. Identify the adjacency state with the neighbor by issuing the show ip ospf neighbor command from privileged EXEC mode.

2. Find the authentication type configured under an interface by issuing the show ip ospf interface command from privileged EXEC mode.

Page 212: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

212

To resolve this issue, perform these steps(cont.): If the authentication type does not match between

two routers on the same segment, you will see a message similar to this:

OSPF: Rcv pkt from x.x.x.x, Ethernet1/0 : Mismatch Authentication type. Input packet specified type 2, we use type 1

This occurs when you issue the debug ip ospf adj command from privileged EXEC mode.

This message indicates this information: Type 0 indicates that no authentication is enabled. Type 1 is for plain text authentication to be enabled. Type 2 means that MD5 authentication is enabled.

Page 213: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

213

To resolve this issue, perform these steps: 3. Make sure that all the routers connected to the sa

me segment use the same authentication type. This is done by issuing the area authentication command in router configuration mode or the ip ospf authentication command in interface configuration mode.

The area authentication command configures all the interfaces under a particular area to use the specified authentication type.

The ip ospf authentication command can be used to individually configure the authentication type for an interface or override the type configured under an interface with the area authentication command.

Page 214: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

214

Trouble shooting IS-IS

© 2002, Cisco Systems, Inc. All rights reserved. 214

Page 215: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

215

Troubleshooting Agenda

Adjacencies LSP Flooding and Contents SPF Computation Monitoring Performance Advanced/New Features

Page 216: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

216

Router-B--------------

interface Loopback0 ip address 192.168.1.1

255.255.255.255 ! Interface Serial0 ip address 192.168.120.10

255.255.255.0 ip router isis ! interface Serial1 ip address 192.168.222.1

255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0001.1921.6800.1001.00

Router-A--------------

interface Loopback0 ip address 192.168.1.5

255.255.255.255 ! interface Serial0 ip address 192.168.120.5

255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0001.1921.6800.1005.00 is-type level-1

Basic Configuration

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 217: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

217

Router-C--------------

interface Loopback0 ip address 192.168.2.2

255.255.255.255 ! interface Serial0 ip address 192.168.111.2

255.255.255.0 ip router isis isis circuit-type level-1 ! interface Serial1 ip address 192.168.222.2

255.255.255.0 ip router isis isis circuit-type level-2 ! router isis passive-interface Loopback0 net 49.0002.1921.6800.2002.00

Router-D--------------

interface Loopback0 ip address 192.168.2.4

255.255.255.255 ! interface Serial1 ip address 192.168.111.4

255.255.255.0 ip router isis ! router isis passive-interface Loopback0 net 49.0002.1921.6800.2004.00 is-type level-1

Basic Configuration

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 218: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

218218218© 2004 Cisco Systems, Inc. All rights reserved.

ADJACENCIES

RST-33029803_05_2004_c2

Page 219: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

219

Check that CLNS Is Running OK?

General CLNS information at a glance: Lists number of CLNS-enabled interfaces Lists NET configured on router Lists the mode that is running (IP or OSI) Lists IS-type

show clns show clns protocol

Page 220: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

220

Rtr-B#show clnsGlobal CLNS Information: 2 Interfaces Enabled for CLNS NET: 49.0001.1921.6800.1001.00 Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64 ERPDU's requested on locally generated packets Running IS-IS in IP-only mode (CLNS forwarding not allowed)

show clns

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 221: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

221

Rtr-B#show clns protocol

IS-IS Router: <Null Tag> System Id: 1921.6800.1001.00 IS-Type: level-1-2 Manual area address(es): 49.0001 Routing for area address(es): 49.0001 Interfaces supported by IS-IS: Serial1 - IP Serial0 - IP Redistribute: static (on by default) Distance for L2 CLNS routes: 110

show clns protocol

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 222: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

222

Are the Hellos Sent?

Interface up/line protocol up Are bits exchanged: sh cdp neigh

detail Show clns neighbor Debug isis adj-packet

Page 223: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

223

Rtr-B#show ip int briefInterface IP-Address OK? Method Status ProtocolLoopback0 192.168.1.1 YES NVRAM up up

Serial0 192.168.222.1 YES manual up up

Serial1 192.168.120.10 YES manual up up

Check Interface Status

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 224: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

224

Rtr-B#show cdp neighborCapability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capability Platform Port IDRtr-C Ser 0 125 R 2500 Ser 1Rtr-A Ser 1 148 R 2500 Ser 0

Check CDP Neighbor Connectivity

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 225: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

225

Are the Adjacencies Up?

Entry in “show clns neighbor”? Protocol is IS-IS (not ES-IS)? State is up (should not show init)? IP addresses on both sides match? MTU size OK?

Page 226: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

226

Rtr-B#show clns neighbors

System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS1921.6800.1005 Se1 *HDLC* Up 21 L1 IS-IS

Check CLNS Neighbors

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 227: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

227

Rtr-B#sh clns nei det

System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS Area Address(es): 49.0002 IP Address(es): 192.168.222.2* Uptime: 03:09:191921.6800.1005 Se1 *HDLC* Up 27 L1 IS-IS Area Address(es): 49.0001 IP Address(es): 192.168.120.5* Uptime: 03:50:01

Check CLNS Neighbor Detail

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 228: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

228

Rtr-B#show clns neighbors

System Id Interface SNPA State Holdtime Type ProtocolRtr-C Se0 *HDLC* Up 23 L2 IS-IS1921.6800.1005 Se1 *HDLC* Up 21 L1 ES-IS

If Protocol Field Shows ES-IS?

Misconfigured IP interface subnet Cisco IOS® validates source IP

address of neighbor before bringing up adjacency

No “ip router isis” command on interface

Common Causes:

Page 229: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

229

Mixing IS-IS IP and IS-IS OSI

Can cause protocol field to show ES-IS!

Mode is per router, not per interface Using passive-interface forces IP mode

All routers in an area must agree IP-only, CLNS-only or integrated (dual)

Can only mix areas, not inside areas L2 routing not guaranteed to work

Page 230: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

230

Rtr-B#show clns int serial1Serial1 is up, line protocol is up Checksums enabled, MTU 1500, Encapsulation HDLC ERPDUs enabled, min. interval 10 msec. CLNS fast switching enabled CLNS SSE switching disabled DEC compatibility mode OFF for this interface Next ESH/ISH in 47 seconds Routing Protocol: IS-IS Circuit Type: level-1-2 Interface number 0x2, local circuit ID 0x101 Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-A.00 Number of active level-1 adjacencies: 1 Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.01 Number of active level-2 adjacencies: 0 Next IS-IS Hello in 6 seconds

show clns interface

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 231: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

231

Rtr-B#show clns int serial0Serial0 is up, line protocol is up Checksums enabled, MTU 1500, Encapsulation HDLC ERPDUs enabled, min. interval 10 msec. CLNS fast switching enabled CLNS SSE switching disabled DEC compatibility mode OFF for this interface Next ESH/ISH in 30 seconds Routing Protocol: IS-IS Circuit Type: level-1-2 Interface number 0x1, local circuit ID 0x100 Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-C.01 Number of active level-1 adjacencies: 0 Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.00 Number of active level-2 adjacencies: 1 Next IS-IS Hello in 6 seconds

show clns interface

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 232: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

232

Rtr-B#debug isis adj-packetsIS-IS Adjacency related packets debugging is onRtr-B#05:45:21: ISIS-Adj: rcvd state UP, old state UP, new state UP05:45:21: ISIS-Adj: Action = ACCEPT05:45:24: ISIS-Adj: Sending serial IIH on Serial0, length 149905:45:26: ISIS-Adj: Rec serial IIH from *HDLC* (Serial1), cir type L1, cir id 00, length 149905:45:26: ISIS-Adj: rcvd state UP, old state UP, new state UP05:45:26: ISIS-Adj: Action = ACCEPT05:45:26: ISIS-Adj: Sending serial IIH on Serial1, length 149905:45:31: ISIS-Adj: Rec serial IIH from *HDLC* (Serial0), cir type L1L2, cir id 01, length 1499

debug isis adj-packets

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 233: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

233

Are the LSPs Flooded?

show isis database Compare sequence numbers Compare checksum or content debug isis update-packets debug isis snp-packets show clns traffic

Page 234: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

LSP FLOODING AND CONTENTS

Page 235: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

235

Rtr-B#show isis database

IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000020 0x0C24 674 1/0/01921.6800.1005.00-00 0x00000023 0x909E 830 0/0/01921.6800.1005.01-00 0x00000017 0xC896 841 0/0/0IS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000024 0x7D98 748 0/0/0Rtr-C.00-00 0x00000028 0x1E01 1128 0/0/0

show isis database

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 236: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

236

Rtr-B#debug isis update-packetsIS-IS Update related packet debugging is on

Rtr-B#05:45:21: ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24E, ht 1199,05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)05:45:21 : ISIS-Upd: LSP newer than database copy05:45:21 : ISIS-Upd: No change05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-0005:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25E, ht 1199 on Serial005:45:21 : ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24F, ht 1199,05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)05:45:21 : ISIS-Upd: LSP newer than database copy05:45:21 : ISIS-Upd: No change05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-0005:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25F, ht 1199 on Serial0

debug isis update-packets

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 237: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

237

Rtr-B#debug isis snp-packetsIS-IS CSNP/PSNP packets debugging is onRtr-B#07:51:59: ISIS-Snp: Build L2 PSNP entry for 1921.6800.2002.00-00, seq 3507:51:59: ISIS-Snp: Sending L2 PSNP on Serial007:53:50: ISIS-Snp: Rec L1 PSNP from 1921.6800.1005 (Serial1)07:53:50: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 31, ht 119707:53:50: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 3107:54:26: ISIS-Snp: Build L1 PSNP entry for 1921.6800.1005.00-00, seq 2F07:54:26: ISIS-Snp: Sending L1 PSNP on Serial107:55:18: ISIS-Snp: Rec L2 PSNP from 1921.6800.2002 (Serial0)07:55:18: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 32, ht 119707:55:18: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 32

debug isis snp-packets

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 238: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

238

Does the LSP Contain ALL Info? LSPs are identified by:

LSP identifier (8 bytes—sysID, n-sel, frag.) Sequence number (higher means newer

LSP) Remaining lifetime (expiration purges LSP) Checksum (if corrupt discard—sender r-tx’s)

show isis database detail! Correct IP prefixes and metrics present? debug isis local-updates

Page 239: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

239

Rtr-B#sh isis dat det 1921.6800.1001.00-00

IS-IS Level-1 LSP Rtr-B.00-00LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000001 0x20D5 1015 1/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 0 IP 192.168.1.1/32 Metric: 10 IP 192.168.222.0/24 Metric: 10 IP 192.168.120.0/24 Metric: 10000 IS-Extended 1921.6800.1005.00IS-IS Level-2 LSP Rtr-B.00-00LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 * 0x00000263 0xAD32 1184 0/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 10000 IS-Extended Rtr-C.00 Metric: 10 IP 192.168.120.0/24 Metric: 0 IP 192.168.1.1/32 Metric: 20 IP 192.168.1.5/32 Metric: 10 IP 192.168.222.0/24

show isis database detail

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 240: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

240

tr-B#sh debugCLNS: IS-IS local updates debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 1Rtr-B(config-if)#shutRtr-B(config-if)#07:59:26: ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)07:59:26 : ISIS-Upd: Building L1 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)07:59:26 : ISIS-Upd: Building L1 LSP07:59:26 : ISIS-Upd: Building L2 LSP07:59:26 : ISIS-Loc: L2 non-summarized metric change (4294967295->20), 192.168.1.507:59:26 : ISIS-Upd: Building L2 LSP

debug isis local-updates

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 241: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

241

Rtr-B#show isis lsp-log Level 1 LSP log When Count Interface Triggers01:50:44 1 CONFIG01:50:35 1 Loopback0 IPUP01:50:28 1 Serial0 IPUP01:50:20 1 Serial1 IPUP01:50:20 1 Serial1 NEWADJ01:50:18 1 ATTACHFLAG01:36:49 1 Loopback0 CONFIG

Level 2 LSP log When Count Interface Triggers01:50:46 1 CONFIG01:50:36 1 Loopback0 IPUP01:50:30 2 Serial0 NEWADJ IPUP01:50:22 1 Serial1 IPUP01:50:10 1 IPIA01:48:21 1 Serial0 DELADJ01:48:16 1 Serial0 NEWADJ01:36:51 1 Loopback0 CONFIG

show isis lsp-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 242: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

242

LSP Checksum

Depending on Layer 2 CRC is not enough, corruption happens in routers and switches

Compute checksum of received LSP and check against checksum inside LSP

If corrupt, silently discard LSP Sender will always retransmit the LSP

Detection of LSP Corruption During Flooding

Page 243: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

243

Does SPF Calculate Correct Routes?

show isis topology debug isis spf-events debug isis spf-statisticsAlso: show ip route isis debug ip routing debug clns routing

Page 244: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

SPF COMPUTATION

Page 245: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

245

Rtr-B#show isis topology

IS-IS paths to level-1 routersSystem Id Metric Next-Hop Interface SNPARtr-B --1921.6800.1005 10 1921.6800.1005 Se1 *HDLC*

IS-IS paths to level-2 routersSystem Id Metric Next-Hop Interface SNPARtr-B --Rtr-C 10 Rtr-C Se0 *HDLC*

show isis topology

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 246: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

246

Rtr-B#debug isis spf-eventsIS-IS SPF events debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 0Rtr-B(config-if)#shutRtr-B(config-if)#07:46:22: ISIS-Spf: L2 LSP 1 (1921.6800.2002.00-00) flagged for recalculationfrom 35F69EE07:46:22: ISIS-Spf: L1 LSP 2 (1921.6800.1001.00-00) flagged for recalculation from360CF3E07:46:22: %LINK-5-CHANGED: Interface Serial0, changed state to administratively down07:46:22: ISIS-Spf: Calculating routes for L2 LSP 1 (1921.6800.2002.00-00)07:46:22: ISIS-Spf: Add 192.168.111.0/255.255.255.0 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.2.2/255.255.255.255 to IP RIB, metric 1007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.2.4/255.255.255.255 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Add 192.168.222.0/255.255.255.0 to IP RIB, metric 2007:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)07:46:22: ISIS-Spf: Aging L2 LSP 1 (1921.6800.2002.00-00), version 51607:46:22: ISIS-Spf: Calculating routes for L1 LSP 2 (1921.6800.1001.00-00)07:46:22: ISIS-Spf: Aging L1 LSP 2 (1921.6800.1001.00-00), version 144

debug isis spf-events

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 247: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

247

Rtr-B#debug isis spf-statisticsIS-IS SPF Timing and Statistics Data debugging is onRtr-B#conf tEnter configuration commands, one per line. End with CNTL/Z.Rtr-B(config)#int serial 0Rtr-B(config-if)#shutRtr-B(config-if)#no shutRtr-B(config-if)#07:49:37: %LINK-3-UPDOWN: Interface Serial0, changed state to up07:49:37: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-C (Serial0) Up, new adjacency07:49:37: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to up07:49:37: ISIS-Spf: Compute L2 SPT07:49:37: ISIS-Spf: Complete L2 SPT,07:49:37: ISIS-Spf: Compute time 0.008/0.008, 2/0 nodes, 1/0 links, 0 suspendsRtr-B(config-if)#07:49:37: ISIS-Spf: Compute L1 SPT07:49:37: ISIS-Spf: Complete L1 SPT,07:49:37: ISIS-Spf: Compute time 0.004/0.004, 2/0 nodes, 1/0 links, 0 suspends

debug isis spf-statistics

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 248: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

248

Is the Change Noticed?

show isis spf-log debug isis spf-trigger show isis lsp-log on generating

router show clns traffic to watch for PRCs

Page 249: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

249

Rtr-B#show isis spf-log

Level 1 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:52 0 1 1 Rtr-B.00-00 NEWLSP02:16:42 0 1 1 Rtr-B.00-00 TLVCODE02:16:32 0 1 2 Rtr-B.00-00 NEWADJ TLVCONTENT02:16:22 8 3 4 Rtr-B.00-00 ATTACHFLAG LSPHEADER

TLVCON TENT02:02:57 4 3 1 Rtr-B.00-00 TLVCONTENT02:01:52 8 3 1 PERIODIC01:46:52 8 3 1 PERIODIC01:31:53 8 3 1 PERIODIC01:16:52 8 3 1 PERIODIC01:01:52 8 3 1 PERIODIC00:46:52 8 3 1 PERIODIC00:31:51 8 3 1 PERIODIC00:16:51 8 3 1 PERIODIC00:01:50 64 3 1 PERIODIC

show isis spf-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 250: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

250

Level 2 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:54 0 1 1 Rtr-B.00-00 NEWLSP02:16:44 0 1 1 Rtr-B.00-00 TLVCODE02:16:34 8 2 3 Rtr-B.00-00 NEWADJ NEWLSP TLVCONTENT02:14:29 8 2 3 Rtr-B.00-00 NEWADJ TLVCONTENT02:14:23 4 2 1 Rtr-C.00-00 TLVCODE02:13:56 8 2 1 Rtr-C.00-00 TLVCONTENT02:02:59 4 2 1 Rtr-B.00-00 TLVCONTENT02:01:54 4 2 1 PERIODIC01:46:54 4 2 1 PERIODIC01:31:54 4 2 1 PERIODIC01:16:54 4 2 1 PERIODIC01:01:54 4 2 1 PERIODIC00:46:53 4 2 1 PERIODIC00:31:53 4 2 1 PERIODIC00:16:53 4 2 1 PERIODIC00:01:53 60 2 1 PERIODIC

show isis spf-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 251: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

251

Rtr-B#debug isis spf-triggersIS-IS SPF triggering events debugging is onRtr-B#07:32:10: ISIS-Spf: L1 SPF needed, periodic SPF, from 0x356C8DC07:32:10: ISIS-Spf: L2 SPF needed, periodic SPF, from 0x356C8DCRtr-B#conf tRtr-B(config)#int serial0Rtr-B(config-if)#isis metric 15Rtr-B(config-if)# ^Z07:38:27: ISIS-Spf: L1 SPF needed, new metric, from 0x3560762Rtr-B(config)#int serial0Rtr-B(config-if)#shutRtr-B(config-if)# ^Z07:39:23: ISIS-Spf: L2, 1921.6800.1001.00-00 TLV contents changed, code 0x207:39:28: ISIS-Spf: L1 SPF needed, L2 attach changed, from 0x357CF3607:39:28: ISIS-Spf: L1, LSP fields changed 1921.6800.1001.00-00

debug isis spf-triggers

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 252: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

252

Monitoring Performance

show proc cpu IS-IS adjacency process(es) Send and receives hellos Manages adjacency database DIS election

Should be low CPU usage (< 1%)

Page 253: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

MONITORING PERFORMANCE

Page 254: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

254

Monitoring Performance

IS-IS update process(es) SPF computation and flooding

CPU usage should fluctuate Don’t worry unless constant > 20% Distributed (CEF) switching should

help

Page 255: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

255

Rtr-B# sh proc cpuCPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 15760 89776 175 0.00% 0.00% 0.00% 0 Load Meter 2 108116 6965 15522 0.08% 0.00% 0.00% 0 Exec 3 1205472 83405 14453 0.00% 0.25% 0.25% 0 Check heaps . . . 47 70580 121119 582 0.00% 0.00% 0.00% 0 CLNS Input 48 14184 79119 179 0.00% 0.00% 0.00% 0 ES-IS Routing 49 205748 206607 995 0.00% 0.00% 0.00% 0 ISIS Adj 50 34596 52216 662 0.00% 0.00% 0.00% 0 ISIS Upd

show proc cpu

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 256: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

256

show clns traffic LSPs sourced indicates stability of IS LSP retransmissions should stay low PRCs can not be checked elsewhere LSP checksum errors are a bad sign Update queue should not stay full Update queue should not drop much

Page 257: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

257

Rtr-B#show clns trafficCLNS: Time since last clear: neverCLNS & ESIS Output: 669, Input: 4773CLNS Local: 0, Forward: 0CLNS Discards: Hdr Syntax: 0, Checksum: 0, Lifetime: 0, Output cngstn: 0 No Route: 0, Discard Route: 0, Dst Unreachable 0, Encaps. Failed: 0 NLP Unknown: 0, Not an IS: 0CLNS Options: Packets 0, total 0 , bad 0, GQOS 0, cngstn exprncd 0CLNS Segments: Segmented: 0, Failed: 0CLNS Broadcasts: sent: 0, rcvd: 0Echos: Rcvd 0 requests, 0 replies Sent 0 requests, 0 repliesESIS(sent/rcvd): ESHs: 0/0, ISHs: 669/660, RDs: 0/0, QCF: 0/0ISO-IGRP: Querys (sent/rcvd): 0/0 Updates (sent/rcvd): 0/0ISO-IGRP: Router Hellos: (sent/rcvd): 0/0ISO-IGRP Syntax Errors: 0

IS-IS: Time since last clear: neverIS-IS: Level-1 Hellos (sent/rcvd): 282/0

show clns traffic

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 258: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

258

IS-IS: Level-2 Hellos (sent/rcvd): 285/0IS-IS: PTP Hellos (sent/rcvd): 420/415IS-IS: Level-1 LSPs sourced (new/refresh): 8/2IS-IS: Level-2 LSPs sourced (new/refresh): 9/1IS-IS: Level-1 LSPs flooded (sent/rcvd): 5/8IS-IS: Level-2 LSPs flooded (sent/rcvd): 7/8IS-IS: LSP Retransmissions: 0IS-IS: Level-1 CSNPs (sent/rcvd): 1/1IS-IS: Level-2 CSNPs (sent/rcvd): 2/2IS-IS: Level-1 PSNPs (sent/rcvd): 7/4IS-IS: Level-2 PSNPs (sent/rcvd): 7/5IS-IS: Level-1 DR Elections: 1IS-IS: Level-2 DR Elections: 1IS-IS: Level-1 SPF Calculations: 7IS-IS: Level-2 SPF Calculations: 9IS-IS: Level-1 Partial Route Calculations: 1IS-IS: Level-2 Partial Route Calculations: 5IS-IS: LSP checksum errors received: 0IS-IS: Update process queue depth: 0/200IS-IS: Update process packets dropped: 0

show clns traffic

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 259: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

259

show isis spf-log

How often are SPFs run? Doesn’t really matter how often Frequent SPFs can indicate a problem In a stable network—only periodic SPF

runs Who triggered the SPF?!

Check the LSPID of first trigger LSP Helps to find the source of the problem

Page 260: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

260

Rtr-B#show isis spf-log

Level 1 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:52 0 1 1 Rtr-B.00-00 NEWLSP02:16:42 0 1 1 Rtr-B.00-00 TLVCODE02:16:32 0 1 2 Rtr-B.00-00 NEWADJ TLVCONTENT02:16:22 8 3 4 Rtr-B.00-00 ATTACHFLAG LSPHEADER

TLVCON TENT02:02:57 4 3 1 Rtr-B.00-00 TLVCONTENT02:01:52 8 3 1 PERIODIC01:46:52 8 3 1 PERIODIC01:31:53 8 3 1 PERIODIC01:16:52 8 3 1 PERIODIC01:01:52 8 3 1 PERIODIC00:46:52 8 3 1 PERIODIC00:31:51 8 3 1 PERIODIC00:16:51 8 3 1 PERIODIC00:01:50 64 3 1 PERIODIC

show isis spf-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 261: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

261

show isis spf-log

Normally in the region of msecs 400 node network takes around 50msecs If closer to seconds then maybe an issue (Could mean LSP re-transmissions over

p2p links) If so, can change lsp-retransmit-timer

How Much CPU Time Is Used per SPF?

Page 262: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

262

Level 2 SPF log When Duration Nodes Count First trigger LSP Triggers02:16:54 0 1 1 Rtr-B.00-00 NEWLSP02:16:44 0 1 1 Rtr-B.00-00 TLVCODE02:16:34 8 2 3 Rtr-B.00-00 NEWADJ NEWLSP TLVCONTENT02:14:29 8 2 3 Rtr-B.00-00 NEWADJ TLVCONTENT02:14:23 4 2 1 Rtr-C.00-00 TLVCODE02:13:56 8 2 1 Rtr-C.00-00 TLVCONTENT02:02:59 4 2 1 Rtr-B.00-00 TLVCONTENT02:01:54 4 2 1 PERIODIC01:46:54 4 2 1 PERIODIC01:31:54 4 2 1 PERIODIC01:16:54 4 2 1 PERIODIC01:01:54 4 2 1 PERIODIC00:46:53 4 2 1 PERIODIC00:31:53 4 2 1 PERIODIC00:16:53 4 2 1 PERIODIC00:01:53 60 2 1 PERIODIC

show isis spf-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 263: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

263

show isis lsp-log

How often do we generate a new LSP?

One router can influence whole net Why did it generate a new LSP? Flapping adjacency is shown by

interface show clns neighbour detail

Look at the “uptime value”

Page 264: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

264

Rtr-B#show isis lsp-log

Level 1 LSP log When Count Interface Triggers01:50:44 1 CONFIG01:50:35 1 Loopback0 IPUP01:50:28 1 Serial0 IPUP01:50:20 1 Serial1 IPUP01:50:20 1 Serial1 NEWADJ01:50:18 1 ATTACHFLAG01:36:49 1 Loopback0 CONFIG Level 2 LSP log When Count Interface Triggers01:50:46 1 CONFIG01:50:36 1 Loopback0 IPUP01:50:30 2 Serial0 NEWADJ IPUP01:50:22 1 Serial1 IPUP01:50:10 1 IPIA01:48:21 1 Serial0 DELADJ01:48:16 1 Serial0 NEWADJ01:36:51 1 Loopback0 CONFIG

show isis lsp-log

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 265: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

265

Traffic Engineering

Support for MPLS-TE: Traffic Engineering with Multi Protocol La

bel Switching IS-IS allows MPLS-TE to flood resourc

e, policy and reservation information about links inside LSPs

New information carried in sub-TLVs IS-IS Metrics were extended for TE

Page 266: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

ADVANCED/NEW FEATURES

Page 267: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

267

Traffic Engineering: IS-IS Metrics

The interface metric was increased from 6 bits wide to 24 bits wide

The total path metric was increased from 10 bits wide to ~32 bits wide

Can configure the old or new metrics Default is old style metrics However—must configure wide

metrics to use TE

Page 268: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

268

Traffic Engineering: IS-IS Metric Styles

Narrow Use old style of TLVs with narrow metric

Wide Use new style of TLVs to carry wider metric

Transition Used when migrating from old to new format Router will accept both old and new style

metrics Caveat: Should only be used in transitioning

Current Metric Styles That May Be Present:

Page 269: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

269

Rtr-C#sh isis dat det

IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-C.00-00 * 0x00000080 0x2237 958 1/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.111.0/24 Metric: 10 IP 192.168.222.0/24 Metric: 10000 IS-Extended Rtr-D.00Rtr-D.00-00 0x00000076 0xB426 1137 0/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-D IP Address: 192.168.2.4 Metric: 0 IP 192.168.2.4/32 Metric: 10 IP 192.168.111.0/24 Metric: 10000 IS-Extended Rtr-C.00

Traffic Engineering Support

sh isis database detail Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 270: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

270

IS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-B.00-00 0x0000007D 0x0BAD 670 0/0/0 Area Address: 49.0001 NLPID: 0xCC Hostname: Rtr-B IP Address: 192.168.1.1 Metric: 6250000 IS-Extended Rtr-C.00 Metric: 10 IP 192.168.120.0/24 Metric: 0 IP 192.168.1.1/32 Metric: 10 IP 192.168.1.5/32 Metric: 10 IP 192.168.222.0/24Rtr-C.00-00 * 0x00000080 0xA130 1141 0/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 10000 IS-Extended Rtr-B.00 Metric: 10 IP 192.168.111.0/24 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.2.4/32 Metric: 10 IP 192.168.222.0/24

Traffic Engineering Support

sh isis database detail Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 271: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

271

hr3.iad.00-00 0x00001FE3 0x5FEE 34751 0/0/0 Area Address: 39.840f.8011.3824.0000.9000.0820 NLPID: 0xCC Hostname: hr3.iad Router ID: 204.152.166.8 IP Address: 204.152.166.8 Metric: 5 IS-Extended hr3.iad.03 Affinity: 0x00000000 Interface IP Address: 209.143.220.163 Physical BW: 100000000 bits/sec Reservable BW: 100000000 bits/sec BW Unreserved[0]: 100000000 bits/sec, BW Unreserved[1]: 100000000 bits/sec BW Unreserved[2]: 100000000 bits/sec, BW Unreserved[3]: 100000000 bits/sec BW Unreserved[4]: 100000000 bits/sec, BW Unreserved[5]: 100000000 bits/sec BW Unreserved[6]: 100000000 bits/sec, BW Unreserved[7]: 100000000 bits/sec Metric: 4 IS-Extended cr2.iad3.00 Affinity: 0x00000000 Interface IP Address: 206.132.253.62 Neighbor IP Address: 206.132.253.61 Physical BW: 155000000 bits/sec Reservable BW: 155000000 bits/sec BW Unreserved[0]: 155000000 bits/sec, BW Unreserved[1]: 155000000 bits/sec BW Unreserved[2]: 155000000 bits/sec, BW Unreserved[3]: 155000000 bits/sec BW Unreserved[4]: 155000000 bits/sec, BW Unreserved[5]: 155000000 bits/sec BW Unreserved[6]: 155000000 bits/sec, BW Unreserved[7]: 155000000 bits/sec

Traffic Engineering Support

sh isis database verbose (for TE BW info)

Page 272: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

272

System ID: wr1.sfo1.00Router ID: 206.132.110.69Link Count: 9 Link[1] Neighbor System ID: wr2.sfo1.00 (P2P link) Interface IP address: 206.132.110.73 Neighbor IP Address: 206.132.110.74 Admin. Weight: 1 Physical BW: 2488000000 bits/sec Reservable BW: 2480000000 bits/sec BW unreserved[0]: 2480000000 bits/sec, BW unreserved[1]: 2420891904 bits/sec BW unreserved[2]: 2420771840 bits/sec, BW unreserved[3]: 2356474880 bits/sec BW unreserved[4]: 2313820928 bits/sec, BW unreserved[5]: 2313820928 bits/sec BW unreserved[6]: 2313820928 bits/sec, BW unreserved[7]: 2313820928 bits/sec Affinity Bits: 0x00000000

Traffic Engineering Support

sh isis mpls traffic-eng advertisementsAlso to View Bandwidth Information Use:

Page 273: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

273

sh isis mpls traffic-eng tunnel

Rtr-B# show isis mpls traffic-eng tunnel Station Id Tunnel Name Bandwidth Nexthop Metric ModeRtr-C.00 Tunnel1022 3333 2.2.2.2 -3 Relative Tunnel1021 10000 2.2.2.2 11 AbsoluteRtr-D.00 Tunnel1031 10000 3.3.3.3 -1 Relative Tunnel1032 10000 3.3.3.3

Traffic Engineering Support

To View Tunnel Information:

Page 274: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

274

Rtr-B#show isis mpls traffic-eng adjacency-log IS-IS RRR logWhen Neighbor ID IP Address Interface Status Level04:52:52 0000.0024.0004.02 0.0.0.0 Et0/2 Up level-104:52:50 0000.0026.0001.00 170.1.1.2 PO1/0/0 Up level-104:52:37 0000.0024.0004.02 0.0.0.0 Et0/2 Up level-1

Traffic Engineering Support

sh isis mpls traffic-eng adjacency-log

To View Adjacency Information:

Page 275: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

275

Rtr-B#debug isis update-packetsSep 2 19:03:19.591: ISIS-Update: Rec L1 LSP 3333.3333.3333.00-00, seq B5D, ht 1199,Sep 2 19:03:19.591: ISIS-Update: from SNPA *HDLC* (Serial2/2)Sep 2 19:03:19.591: ISIS-Update: LSP newer than database copySep 2 19:03:19.591: ISIS-Update: TLV code mismatch (87, 80)Sep 2 19:03:19.591: ISIS-Update: TLV contents different, code 87Sep 2 19:03:19.591: ISIS-Update: TLV code mismatch (16, 2)Sep 2 19:03:19.591: ISIS-Update: Full SPF required

Metric Style Issues Metric style issues An IS is configured for WIDE metrics and receives a TLV

with NARROW metrics: Mismatch TLVs 0x80 (TLV 128) and 0x87 (TLV 135) TLV 128 is IP Internal Reachability (Narrow) TLV 135 is Extended IP Reachability TLV (Wide)

Page 276: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

276

Route Leaking

Feature to enable redistributing Level 2 IP routes into Level 1 areas

Enables Level 1-only routers to pick the best path to exit the area

Enables shortest-exit and MED for BGP Redistribution can be controlled via distr

ibute-lists IP-only feature (CLNS still uses stub)

Page 277: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

277

Rtr-D#sh ip ro isisi ia 1.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 2.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 3.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 4.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 5.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 6.0.0.0/8 [115/74] via 192.168.111.2, Serial1i ia 7.0.0.0/8 [115/74] via 192.168.111.2, Serial1 192.168.2.0/32 is subnetted, 2 subnetsi L1 192.168.2.2 [115/10] via 192.168.111.2, Serial1i L1 192.168.222.0/24 [115/20] via 192.168.111.2, Serial1i*L1 0.0.0.0/0 [115/10] via 192.168.111.2, Serial1

Routes highlighted are shown as IS-IS inter-area Routes originated on Rtr-B and leaked from Rtr-C

Verifying Route Leaking

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Leaking from Rtr-C to Rtr-D:

Page 278: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

278

Rtr-D#sh isis dat det

IS-IS Level-1 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-C.00-00 0x000000FB 0x8E85 620 1/0/0 Area Address: 49.0002 NLPID: 0xCC Hostname: Rtr-C IP Address: 192.168.2.2 Metric: 0 IP 192.168.2.2/32 Metric: 10 IP 192.168.111.0/24 Metric: 10 IP 192.168.222.0/24 Metric: 10000 IS-Extended Rtr-D.00 Metric: 74 IP-Interarea 1.0.0.0/8 Metric: 74 IP-Interarea 2.0.0.0/8 Metric: 74 IP-Interarea 3.0.0.0/8 Metric: 74 IP-Interarea 4.0.0.0/8 Metric: 74 IP-Interarea 5.0.0.0/8 Metric: 74 IP-Interarea 6.0.0.0/8 Metric: 74 IP-Interarea 7.0.0.0/8

Verifying Route Leaking

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Leaking from Rtr-C to Rtr-D:

Page 279: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

279

HMAC-MD5 Authentication Supported platforms:

12000, 10720, 10000, 7500, 7200 Supported across all packet types:

Hellos, LSPs, CSNPs, PSNPs Can authenticate under the IS-IS process

(for LSP/CSNP/PSNP) and/or on the interface(for hellos)

Can authenticate at Level 1 and/or Level 2

Page 280: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

280

!key chain cisco key 100 key-string systems!interface Serial1/0 ip address 10.1.1.1 255.255.255.252 ip router isis isis authentication mode md5 isis authentication key-chain cisco!router isis net 49.0000.0101.0101.0101.00 authentication mode md5 authentication key-chain cisco

HMAC-MD5 Authentication

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Typical Configuration:

Page 281: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

281

Rtr-A#sh key chain

Key-chain cisco:

key 100 -- text "systems"

accept lifetime (always valid) - (always valid) [valid now]

send lifetime (always valid) - (always valid) [valid now]

Rtr-A#sh clns nei

System Id Interface SNPA State Holdtime Type Protocol

Rtr-B Se2/2 *HDLC* Up 28 L1L2 IS-IS

HMAC-MD5 Authentication

Use “sh clns neighbor” to verify adjacency

Use “sh key chain” to view key chain information

Page 282: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

282

Rtr-A#debug isis authentication informationSep 23 15:41:45.040: ISIS-AuthInfo: IIH no change, use the same hmac valueSep 23 15:41:54.880: ISIS-AuthInfo: IIH no change, use the same hmac valueSep 23 15:42:04.620: ISIS-AuthInfo: IIH no change, use the same hmac value

HMAC-MD5 Authentication Use “debug isis authentication information” Debug below shows normal operation as same

MD5 info is re-used if there is no change in IIH

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 283: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

283

Rtr-B#debug isis authentication informationSep 23 15:42:28.950: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:29.830: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.106: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.422: ISIS-AuthInfo: No auth TLV found in received packetSep 23 15:42:30.458: ISIS-AuthInfo: No auth TLV found in received packet

HMAC-MD5 Authentication Use “debug isis authentication information” Debug below shows that local router received

a packet that does not contain authentication data

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 284: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

284

Rtr-A#sh isis datIS-IS Level-1 Link State DatabaseLSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-A.00-00 * 0x00000002 0xDFE8 1195 0/0/0

IS-IS Level-2 Link State DatabaseLSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OLRtr-A.00-00 * 0x00000002 0x5E9A 1195 0/0/0

HMAC-MD5 Authentication Neighbor does not use authentication for the

IS-IS instance No LSPs, CSNPs or PSNPs sent with

authentication No LSPs, CNPs or PSNPs received by

neighbor

Page 285: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

285

Rtr-A#sh ip route

192.168.60.0/30 is subnetted, 1 subnetsC 192.168.60.4 is directly connected, FastEthernet0/0 192.168.110.0/32 is subnetted, 1 subnetsC 192.168.110.1 is directly connected, Loopback0 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial2/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet5/0

HMAC-MD5 Authentication

No neighbor routes installed in local RIB

Page 286: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

286

Rtr-A#Sep 23 17:16:36.191: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-B (Serial2/2) Down, hold time expired

Rtr-A#sh clns neiSystem Id Interface SNPA State Holdtime Type Protocol

HMAC-MD5 Authentication Neighbor does not use authentication for the

interface No hellos sent with authentication No hellos received by neighbor Neighborship is taken down—hold time

expired

Page 287: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

287

Rtr-B#Sep 23 17:16:43.102: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-A (Serial0/2) Down, neighbor forgot us

Rtr-B#sh clns neiSystem Id Interface SNPA State Holdtime Type ProtocolRtr-A Se0/2 *HDLC* Init 27 L1L2 IS-IS

HMAC-MD5 Authentication

Neighbor does not use authentication for the interface Neighborship taken down Non-authenticating router in Init state

Page 288: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

288

Rtr-A#sh ip ro 192.168.60.0/30 is subnetted, 1 subnetsC 192.168.60.4 is directly connected, FastEthernet0/0 192.168.110.0/32 is subnetted, 1 subnetsC 192.168.110.1 is directly connected, Loopback0 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial2/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet5/0

HMAC-MD5 Authentication

Rtr-B#sh ip ro 192.168.30.0/30 is subnetted, 1 subnetsC 192.168.30.4 is directly connected, Serial0/0 192.168.150.0/30 is subnetted, 1 subnetsC 192.168.150.4 is directly connected, Ethernet3/2 192.168.20.0/30 is subnetted, 1 subnetsC 192.168.20.12 is directly connected, Serial0/2 10.0.0.0/22 is subnetted, 1 subnetsC 10.200.96.0 is directly connected, Ethernet3/0 192.168.220.0/32 is subnetted, 1 subnetsC 192.168.220.1 is directly connected, Loopback0

No Routes in RIB on Both Neighbors

Page 289: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

289

Multi-Topology (MT) Support for IS-IS Now IPv4 and IPv6 topologies are

separate No need for IPv4 and IPv6 to be congruent

Lifts previous restrictions Much easier deployment of IPv6

Avoids previous possible complications Supported from 12.0(26)S, 12.2(15)T Draft-ietf-isis-wg-multi-topology-xx.txt

Page 290: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

290

MT for IPv6: Hello Processing

Basic hello processing and checking is the same MT membership is advertised in IIH packets

MT ID #0 for IPv4 MT ID #2 for IPv6

Must have at least one common set of topology types to form an adjacency

LAN nodes will always establish an adjacency regardless of MT for reliable flooding and sync

Maintaining MT Adjacencies

Page 291: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

291

MT for IPv6: LSP Generation and Flooding

The LSP flooding mechanism is unchanged for multi-topology integrated IS-IS

For LANs—DIS, CSNP and PSNP functions are unchanged by MT extension

Use standard show and debug commands to troubleshoot

Page 292: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

292

MT for IPv6: SPF and RIBs Each topology (IPv4 and IPv6) runs its own SPF During IPv6 SPF we examine TLV 222 for MT ID

#2 to build the Shortest Path Tree Two-Way Connectivity Check (TWCC) follows TWCC ensures bidirectional reachability

For leaf nodes we examine TLV 237 for MT ID #2 and install the IPv6 prefixes in the IPv6 RIB

Page 293: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

293

MT for IPv6: Memory and Performance

Additional memory will be required However—will not have major impact on

overall system requirements Performance impact will be additional

SPF runtime to compute L1 and L2 IPv6 topology However IPv6 SPF not run back-to-back

with IPv4—but interleaved so no CPU hogging

Page 294: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

294

MT for IPv6: Restrictions Not compatible with previous implementation

Not compatible with single SPF IPv6 However transition mode possible to migrate

from current IPv6 to MT IPv6 Advertises both IPv6 and MT IPv6 TLVs

If IPv4 and IPv6 are configured on the same interface—they must be running the same level

Must use wide metrics

Page 295: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

295

Multi-Topology IS-IS Example

IPv4-IPv6 Enable Router

Area A

Area BArea C

Area D

IPv4-Only Enable Router

The Multi-Topology Software Will Create Two Topologies Inside Area IPv4 and IPv6

IPv4-Only Routers Will Be Excluded from the IPv6 Topology

Page 296: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

296

Area B

MT for IPv6: Basic Configuration

The optional keyword TRANSITION may be used for transitioning existing IS-IS IPv6 single SPF mode to MT IS-IS

Wide metric is mandated for Multi-Topology to work

Router1# interface ethernet-1 ip address 10.1.1.1 255.255.255.0 ipv6 address 2001:0001::45c/64 ip router isis ipv6 router isis isis ipv6 metric 20

interface ethernet-2 ip address 10.2.1.1 255.255.255.0 ipv6 address 2001:0002::45a/64 ip router isis ipv6 router isis isis ipv6 metric 20

router isisnet 49.0000.0100.0000.0000.0500metric-style wide!address-family ipv6multi-topology exit-address-family

LAN1: 2001:0001::45c/64

LAN2: 2001:0002::45a/64

Ethernet-1

Ethernet-2

Router1

Page 297: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

297

Cisco IOS Multi-Topology IS-IS Display

Router# show clns neighbors detailSystem Id Interface SNPA State Holdtime Type Protocol2653 Se0/1 *HDLC* Up 25 L1L2 M-ISIS

Area Address(es): 49.0000.01IP Address(es): 192.168.0.6*IPv6 Address(es): FE80::204:C1FF:FEDB:2FA0Uptime: 00:01:22Topology: IPv4, IPv6

2652# show isis database detailIS-IS Level-2 Link State Database:LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL2651.00-00 0x0000000F 0x0161 1066 0/0/0

Area Address: 49.0000.01Topology: IPv4 (0x0) IPv6 (0x2)NLPID: 0xCC 0x8EHostname: 2651IP Address: 192.168.0.2IPv6 Address: 3FFF:FFFF:2::1Metric: 10 IS-Extended 2652.00Metric: 10 IS-Extended 2653.01Metric: 10 IS (MT-IPv6) 2653.01Metric: 10 IP 192.168.0.0/30Metric: 20 IP 192.168.0.4/30Metric: 10 IP 192.168.1.0/24Metric: 20 IPv6 (MT-IPv6) 3FFF:FFFF:1::/64Metric: 10 IPv6 (MT-IPv6) 3FFF:FFFF:2::/64

MT IS-IS

Page 298: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

298

Local RIB

The Local RIB contains entries for all possible routes to destinations

From this RIB, the best route is chosen to be installed in the ‘IP routing table’

Local RIB is also necessary for prefix prioritization

Local RIB can significantly reduceconvergence time

Page 299: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

299

Rtr-B#sh isis rib ? A.B.C.D Network prefix redistribution ISIS IP redistribution RIB information | Output modifiers <cr>

Rtr-B#sh isis rib 192.168.40.14 IPv4 local RIB for IS-IS process Routes under majornet 192.168.40.0/24:192.168.40.4/30 [115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51] [115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166] 192.168.40.16/30 [115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51] [115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166]

Viewing Local RIB Information

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 300: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

300

Rtr-B#debug isis rib ? global IS-IS IP routes in global RIB local IS-IS IP routes in local RIB redistribution IS-IS IP redistribution RIB

Rtr-B#debug isis rib local IS-IS IPv4 local RIB debugging is onRtr-B#*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: RT exists*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.17 tg 0 LSP[3/(51->52)]*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Create new RT*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: create new path: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.5 tag 0 LSP[3/52]*Mar 15 01:18:34.882: ISIS-LR: Enqueued to updateQ[2] for 192.168.40.12/30*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: Looking for RT *Mar 15 01:18:34.882: ISIS-LR: RT exists*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20] via 192.168.40.5(Se0/0) from 192.168.40.17 tg 0 LSP[3/(51->52)]*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: not aged out in LSP ix 3 same ver(52)*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: not aged out in LSP ix 3 same ver(52*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: not aged out in LSP ix 3 same ver(52)

Debugging Local RIB Information

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 301: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

301

Rtr-B#debug isis rib ? global IS-IS IP routes in global RIB local IS-IS IP routes in local RIB redistribution IS-IS IP redistribution RIB

Rtr-B#debug isis rib globalIS-IS IPv4 global RIB debugging is onRtr-B# *Mar 15 01:22:06.117: ISIS-GR: ------ Start updateQ, from 60773650*Mar 15 01:22:06.117: ISIS-GR: 192.168.210.1 255.255.255.255: updateQ[1] entry*Mar 15 01:22:06.117: ISIS-GR: del rdb *Mar 15 01:22:06.117: ISIS-GR: 192.168.250.12 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.117: ISIS-GR: del rdb *Mar 15 01:22:06.117: ISIS-GR: 192.168.250.4 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: 192.168.30.16 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: 192.168.20.4 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.121: ISIS-GR: del rdb *Mar 15 01:22:06.121: ISIS-GR: ------ End updateQ, 0 suspends*Mar 15 01:22:06.121: ISIS-GR: ------ Start updateQ, from 60773DE0*Mar 15 01:22:06.121: ISIS-GR: 192.168.40.12 255.255.255.252: updateQ[2] entry*Mar 15 01:22:06.125: ISIS-GR: del rdb *Mar 15 01:22:06.125: ISIS-GR: ------ End updateQ, 0 suspends

Debugging Global RIB Information

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 302: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

302

Incremental SPF (i-SPF)

Modified Dijkstra algorithm We keep the unchanged part of the tree We rebuild only the affected parts of

the tree Re-attach the affected parts of the tree

to the unchanged part of the tree Fairly complex algorithm

Incremental SPF

Page 303: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

303

Incremental SPF (i-SPF) When new LSPs are received, each router will

check what has changed in the LSP Based on the changed information, the SPT is

“modified” in order to reflect the changes Start the computation from the node that

received the change i-SPF is essentially used to prepare lists for

running standard Dijkstra and computing next-hops

No need to run SPF if a link that was not in the SPT is reported down

Page 304: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

304

S0

S1

S2S3

D

C-G Link Is Down; C-G Link Was Not Used in SPT Anyway, Therefore There Is No Need to Run SPF

Incremental SPF (i-SPF)

Cost: 0, NH: --A

Cost: 3, NH: BB

Cost: 11, NH: BE

Cost: 13, NH: DG

Cost: 6, NH: D, BC Cost: 3, NH: D

Cost: 8, NH: D, BF

Page 305: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

305

S0

S1

S2S3

F Reports a New Neighbor; the SPT Need Only To Be Extended Behind F; There Is No Need for Router A to Recompute the Whole SPTRouter A Will Compute SPF from Node F

H

Incremental SPF (i-SPF)

Cost: 0, NH: --A

Cost: 3, NH: BB

Cost: 11, NH: BE

Cost: 13, NH: DG

Cost: 6, NH: D, BC Cost: 3, NH: DD

Cost: 8, NH: D, BF

Page 306: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

306

Incremental SPF (i-SPF): Configuration When router is powered on—we always run a

full SPF to build the tree To configure i-SPF under router isis:

ispf [level-1/level-2/level-1-2] [1-600] Must specify levels The last parameter (1–600) is optional and

corresponds to the duration (in seconds) of full SPF runs after reload

By default, full SPF is run for the first 60 seconds after reload—even if i-SPF is configured

Integrated into 12.0(24)S

Page 307: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

307

Rtr-A#debug isis spf-statistics*Mar 15 01:03:44.044: ISIS-Stats: Compute L1 SPT*Mar 15 01:03:44.044: ISIS-Stats: Starting incremental SPF for level-1*Mar 15 01:03:44.048: ISIS-Stats: SPF only compute time 0.000*Mar 15 01:03:44.048: ISIS-Stats: IPv4 RIB only compute time 0.004*Mar 15 01:03:44.048: ISIS-Stats: Complete L1 SPT, Compute time 0.004, 2 nodes, 0 links on SPT, 0 suspends*Mar 15 01:03:44.048: ISIS-Stats: I-SPF NewLSP: 0.000 (3 nodes) - Reattach: 0.000 - WalkTENT: 0.000*Mar 15 01:03:44.048: ISIS-Stats: I-SPF WalkParents: 0.000 (0 nodes) - Total time: 0.000

debug isis spf-statistics

L1 Routers

Area 49.0001

Rtr-A

Rtr-B S0

S1

S0

L1L2 Routers

Area 49.0002

Rtr-D

Rtr-C S0

S1

S1

Page 308: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

308

Rtr-A#sh isis spf-log detail

level 1 SPF log

i-SPF triggered 2 times, first at 01:03:38.548 Mar 15 1993 by TLVCONTENT wait enforced 5.500, next wait interval 5.500 SPT node total/processed/time: 4/2/0.000 RIB prefix processed/time: 0/0.000 1/0.000 3/0.004

Full SPF triggered 2 times, first at 00:51:17.877 Mar 15 1993 by NEWADJ wait enforced 5.500, next wait interval 5.500 SPT node total/processed/time: 2/2/0.000 RIB prefix processed/time: 0/0.000 0/0.000 2/0.004

Viewing i-SPF

Area 49.0001

Rtr-A

Rtr-B

S0

S1

S0

L1 Router

L1L2 Routers

Area 49.0002

Rtr-C

Rtr-D

S1

S0

S1

L1 Router

Page 309: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

309

Non-Stop Forwarding Allows uninterrupted data forwarding during a swi

tchover to a standby RP Routing protocol mechanisms are also applicab

le to a restart in the RP Forwarding Information Base (FIB) is maintained a

nd updated once the routing protocolsreconverge

May be used for planned and unplanned events Two modes of NSF available:

Cisco and IETF

Page 310: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

310

NSF: Routing Protocol Requirements Switchover MUST be completed before dead/hold

timer expires Peers will reset the adjacency and reroute the

traffic after that time FIB MUST remain unchanged during switchover

Current routes marked as “dirty” during restart; “cleaned” once convergence is complete

Adjacencies MUST NOT be reset when switchover is complete Protocol state is not maintained

Peers of restarting router SHOULD also beNSF-aware

Page 311: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

311

NSF Configuration Commands

nsf [cisco/ietf ] Enables isis nsf, OFF by default

nsf interval xxx Minimal time interval between two restart (default =5mi

n) nsf holdtime [manual <seconds> | adjacency]

IETF version only Time NSF will wait for the LSP database to synchronize befo

re generating and flooding its own LSP with the overload-bit set

To Configure NSF under “router isis”:

Page 312: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

312

NSF Configuration and Debug Commands nsf interface wait xxx

Cisco version only (default=10 sec, range [1–60] ) Time an NSF restart will wait for all interfaces with ISIS adjacen

cies to come up before completing the restart Show commands

show isis nsf nsf capability and restart information

show clns neighbor detail peer’s nsf capability show isis database detail LSP information

Debug commands debug isis nsf [cisco | detail | ietf] debug isis adj-packets

Page 313: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

313

IS-IS NSF “show commands”

NSF Cisco mode—check that it is enabled

chi-hr1#show isis nsf

NSF is ENABLED, mode 'cisco'

RP is ACTIVE, standby ready, bulk sync completeNSF interval timer expired (NSF restart enabled)Checkpointing enabled, no errorsLocal state: ACTIVE, Peer state: STANDBY HOT, Mode: SSO

We maintain the IS-IS Neighbor and Database Information on the STANDBY [G]RPGRP-Slot9#show clns neighbor

System Id Interface SNPA State Holdtime Type Protocolchi-ar1 PO2/0 *HDLC* Stby 30 L2 IS-IS

Page 314: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

314

chi-hr1#sh isis nsf

NSF is ENABLED, mode 'ietf'NSF pdb state: InactiveNSF L2 active interfaces: 0NSF L2 active LSPs: 0NSF interfaces awaiting L2 CSNP: 0Awaiting L2 LSPs:NSF T3 remaining: 0 secondsInterface: Ethernet0 NSF L2 Restart state: Running NSF L2 Restart retransmissions: 0 Maximum L2 NSF Restart retransmissions: 3Interface: POS2/0 NSF L2 Restart state: Running NSF p2p Restart retransmissions: 0 Maximum L2 NSF Restart retransmissions: 3

IS-IS NSF “show commands”

NSF IETF mode—check that it is enabled

Page 315: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

315

IS-IS NSF “show commands”

chi-hr1#show clns neighbor detailSystem Id Interface SNPA State Holdtime Type Protocolchi-ar1 PO2/0 *HDLC* Up 21 L2 IS-IS Area Address(es): 10 IP Address(es): 172.1.1.21* Uptime: 01:52:02 NSF capable

Page 316: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

316

IS-IS NSF “debug commands”

Rtr-A#debug isis nsf? cisco Include only Cisco NSF information detail Include detailed information ietf Include only IETF NSF information

Other debug information also useful with NSF: debug isis update-packets debug isis snp-packets debug isis adj-packets

Page 317: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

317

IS-IS NSF “debug isis nsf ietf”

*Aug 12 17:32:45.507 PDT: ISIS-NSF: Inserting p2p NSF REQ on POS3/0 IIH

*Aug 12 17:32:45.519 PDT: ISIS-NSF: POS3/0 level-2 state progression: state=Restarting/event=RA Rcvd/newstate=RA Seen

*Aug 12 17:32:45.519 PDT: ISIS-NSF: ISIS NSF restart ACK option received on p2p itf from 2002.0020.0001 (POS3/0)

*Aug 12 17:32:45.559 PDT: ISIS-NSF: POS3/0 level-2 state progression: state=RA Seen/event=CSNP Rcvd/newstate=Running

Page 318: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

318

.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 P2P sync CSNP on PO2/0

.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 CSNP on POS2/0

.Aug 23 14:10:18.926 PDT: ISIS-SNP: Rec L2 PSNP from 2002.0020.0001 (POS2/0)

.Aug 23 14:10:18.926 PDT: ISIS-SNP: PSNP entry 2002.0020.0003.FE-FE, seq 0, ht 1199

.Aug 23 14:10:20.962 PDT: ISIS-Update: Sending L2 LSP 2002.0020.0003.00-00, seq F7, ht 1199 on POS2/0

.Aug 23 14:10:21.790 PDT: ISIS-Adj: Rec serial IIH from *HDLC* (POS2/0), cir type L2, cir id 00, length 4469

.Aug 23 14:10:21.790 PDT: ISIS-Adj: rcvd state UP, old state UP, new state UP

• debug isis update-packets

• debug isis snp-packets

• debug isis adj-packets

IS-IS NSF: Other Useful Debug Commands

Page 319: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Trouble Shooting CEF

Page 320: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

320

Vocabulary

CEFCEF: Cisco Express Forwarding FIBFIB: Forwarding Information Base

collection of data used to make switching decision

• RIBRIB: Routing Information Basetopology information that the router learns from routing protocols

Page 321: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

321

CEF...What is it?

Advanced layer-3, IPIP switching method; scaleable, distributed, high performance

CEF differs from current fast switchingfast switching in the way the router maintains its forwarding table

CEF switching is required for most of the new IP QoS features

Page 322: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

322

Features

Load balancing Per destination (the default) and per

packet over equal/unequal cost links for as many paths as known in the routing topology

Traffic statistics Byte and packet counts at a

granularity of per-prefix, per-neighbor etc..

Page 323: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

323

Features (cont..)

Media independence CEF currently supports Packet over Sonet,

ATM/AAL5, Frame Relay, Ethernet, FDDI, HDLC, PPP and tunnels.

Tunneling: Generic Route Encapsulation (GRE).

Subinterface support: allowing for the flexibility of per subinterface configurations e.g. MTU.

Page 324: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

324

Today’s Problems and CEF’s Solutions

Type of traffic - EfficiencyToday’s traffic is short lived. Cache on demand (fast switching) requires constant cache create/invalidate (aging). FIB has a forwarding entry for every routable prefix.

Dynamic networks - PerformanceThe FIB table mirrors the routing table exactly - thus fast convergence

Page 325: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

325

Today’s Problems and CEF’s Solutions (cont.)

Scalability Core routers carry high number of prefixe

s (mostly recursive) which need to be resolved each time a packet has to be process switched (cache on demand). In CEF, the recursive route is resolved when the FIB table is built. dCEF is also available.

Better use of memory and CPU.

Page 326: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

326

Possible problem The throughput of the router is less th

an expected. The throughput of an interface is less

than expected. Packets are getting dropped when CE

F/dCEF is enabled Packets are not CEF switched but

fast switched instead.

Page 327: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

327

Possible problem(cont.) Packets are not dCEF switched but fast swit

ched instead Packets are not dCEF switched but CEF swit

ched instead. Load-sharing is not working The route processor runs out of memory wh

en dCEF is enabled CEF disables itself on a linecard due to lack

of memory

Page 328: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

The throughput of the router is less than expected.

Possible problem :CEF/dCEF is not enabled globally on the rou

ter.

Page 329: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

329

Solution Step 1 Check is CEF/dCEF is enabled globally on ro

uter. Use global command ip cef for central processor s

witching or ip cef distributed if you want distributed CEF switching. When CEF/dCEF is enabled globally, CEF/dCEF is automatically enabled on the interfaces.

If show ip summary command gives an output that means CEF is enabled. If not, you get the following output for show ip summary:

router#show ip summary %CEF not running

Page 330: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

The throughput of an interface is less than expected.

Possible problem : CEF/dCEF is not enabled on this interface.

Page 331: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

331

Solution Step 1 Check if CEF/dCEF is enabled on the interfa

ce. Use interface command ip route-cache cef for central processor switching or ip route-cache distributed if you want distributed CEF switching.

The result of show cef interface x/x and show ip interface x/x must report one of the following:

IP CEF switching enabled IP Normal CEF switching turbo vector IP Distributed CEF switching enabled IP Distributed CEF switching turbo vector

Below is a example on a router:

Page 332: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

332

Examplerouter#show cef interface FastEthernet0/1 FastEthernet0/1 is up (if_number 3) Corresponding hwidb fast_if_number 3 Corresponding hwidb firstsw->if_number 3 Internet address is 10.254.0.200/24 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set IP policy routing is disabled Hardware idb is FastEthernet0/1 Fast switching type 1, interface type 18 IP CEF switching enabled IP Feature Fast switching turbo vector IP Feature CEF switching turbo vector Input fast flags 0x0, Output fast flags 0x0 ifindex 2(2) Slot 0 Slot unit 1 VC -1 Transmit limit accumulator 0x0 (0x0) IP MTU 1500

Page 333: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are getting dropped when CEF/dCEF is enabled

Possible problem : The CEF entries and the adjacencies

table are not completely built.

Page 334: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

334

Solution Step 1 Verify whether the router has completely built the CEF

entries and adjacencies table. If it is not the case the router may switch the packet using an alternative switching method. This is why the packets are 50% successful. To verify this, issue the command show ip cache. There should be an IP cache entry built for this prefix.

router#show ip cache IP routing cache 0 entries, 0 bytes 3 adds, 3 invalidates, 0 refcounts The above result of the command shows that the router has

been doing cache switching for a moment: 3 adds, 3 invalidates. The result of the ping is 50% successful as the number of packets that are punted is rate limited.

Page 335: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

335

Solution(cont.) router #ping ip Target IP address: 12.0.0.1Repeat count [5]: 20 Datagram size [100]: Timeout in seconds [2]: Extended commands [n]: Sweep range of sizes [n]: Type escape sequence to abort.Sending 20, 100-byte ICMP Echoes to 12.0.0.1, timeout is 2 seconds:!.!.!.!.!.!.!.!.!.!.Success rate is 50 percent (10/20), round-trip min/avg/max = 1/1/1 msAn adjacency on interface POS8/0/0 is still incomplete. router#show adjacencyProtocol Interface AddressIP POS8/0/0 point2point(3)IP Serial10/0/0:28 point2point(3)IP POS8/0/0 point2point(2) (incomplete)

Page 336: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are getting dropped when CEF/dCEF is enabled

Possible problem : Part of the adjacencies is “incomplete adjacency” or “drop adjacency”.

Page 337: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

337

Solution Step 1: Check the features you have enabled; som

e platforms only support dCEF with a limited number of features. If you enable a feature not supported by dCEF a drop adjacency is created and the packets are dropped. Issue the command show cef drop and look for packets being dropped. Below you will find the output of the command when a feature is not supported.

router#show cef drop CEF Drop Statistics Slot Encap_fail Unresolved Unsupported No_r

oute No_adj RP 0 0 2 0 0

Page 338: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

338

Solution(cont.) Step 2 If your hardware supports CEF and dCEF, in case of pa

cket drops you may see only “incomplete adjacency”. Regardless of the platform, here is the method to follow to isolate the problem.

Use show ip route to determine which inbound interface the traffic should be switched through. Issue show ip cef <prefix> and look for the interface adjacencies:

router#show ip cef 10.254.0.200 10.254.0.200/32, version 18, cached adjacency 10.254.0.200 0 packets, 0 bytes via 10.254.0.200, FastEthernet0/1, 0 dependencies next hop 10.254.0.200, FastEthernet0/1 valid cached adjacency

Page 339: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

339

Solution(cont.)router#show adjacency FastEthernet 0/1 detailProtocol Interface AddressIP FastEthernet0/1 10.254.0.200(5) 0 packets, 0 bytes 00055FAF2C06 00055FAC18010800 ARP 03:45:10 IP FastEthernet0/1 server(5) (10.254.0.1) 0 packets, 0 bytes 00508B121E46 00055FAC18010800 ARP 03:54:27

Page 340: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are getting dropped when CEF/dCEF is enabled

Possible problem : None of the above solutions are applying

– try to debug the issue.

Page 341: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

341

Solution Step 1: Check if the packets are dropped or punt fo

r unexpected reason. The following counters are reporting the punt and dropped packets:

show cef not-cef-switched (punt reason counters)

show cef drop (drop reason counters) The above is particularly useful in a controlled envi

ronment. You send a number of packets. If all the packets are dropped or punt you it gives you the reason for this unexpected behavior.

Page 342: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

342

Solution(cont.) Step 2: If the traffic to a destination is limite

d, you can try the following debug commands:

debug ip cef drop <acl> debug ip cef receive <acl> access-list <acl> permit <destination> The drop and receive debug information wil

l provide more information on why packets are being dropped or punted by CEF/dCEF.

Page 343: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are getting dropped when CEF/dCEF is enabled

Possible problem :None of the above solutions are applying –

Call the TAC

Page 344: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

344

Solutionstep 1: As always, for a CEF/dCEF unexpected behavior, collect: show tech cef Step 2: If you cannot get a destination and if none of the above solutions ar

e applying to your case, you will have to collect a list of information to help the Cisco TAC solving your problem.

show ip route <destination> show ip cef <destination> internal show adjacency <output interface > show ip interface brief show cef interface exec all show ip cef <destination> internal exec all show adjacency <output interface> detail exec all show cef interface show ip arp (or analogous link info, e.g. show frame-relay map)To get to a destination, there has to be a route, a forwarding entry, and a val

id adjacency. The data collected above will show which element is missing.

Page 345: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are not CEF switched but fast switched instead.

Possible problem: CEF does not support a configured

feature.

Page 346: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

346

Solution Step 1 While CEF supports a large range of f

eatures, however some features are not supported. If a feature is not supported packets are forwarded to the next slower level of switching - fast switching.

Check the following document to know exactly what is supported:

IOS features supported by CEF and dCEF matrix

Page 347: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

347

Solution(cont.) Step 2 Check the configuration of the router to make

certain CEF is activated. Also use the following commands:

show cef interface x/y, show ip cef <prefix> , show adjacency <interface> detail Ultimately issue show cef not-switch and look for p

ackets that are not CEF switched. If CEF is not able to forward packets to a destination,

the CEF table is built anyway but the adjacency is called a “punt”. If just a part of the packets cannot be CEF switched a cache adjacency is created.

Page 348: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

348

Solution(cont.)Router #show ip cef 172.20.1.2172.20.1.0/30, version 4, attached, connected, per-destination sharing0 packets, 0 bytes via Serial4/0, 1 dependency valid punt adjacencyAs the result of punt adjacency the router will report CEF packets passed on to next switching

layer.router#show cef interface serial4/0 Serial4/0 is up (if_number 13) Corresponding hwidb fast_if_number 13 Corresponding hwidb firstsw->if_number 13 Internet address is 172.20.1.1/30 ICMP redirects are always sent Per packet load-sharing is disabled IP unicast RPF check is disabled Inbound access list is not set Outbound access list is not set IP policy routing is disabled Interface is marked as point to point interface Packets switched to this interface are dropped to the next slow path

Page 349: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

349

Solution(cont.)Step3 Look under interface/global configuration

for features that you are not sure to be compatible with CEF. Log messages from router, then remove and put back one by one each of these features.

When removing a not CEF switched feature, the log will report:

%FIB-5-NOPUNTINTF: CEF resuming switching packets to <interface>

When adding a feature which cannot be CEF switched, the log will report:

%FIB-4-PUNTINTF: CEF punting packets switched to <interface> to next slower path

Page 350: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are not dCEF switched but fast switched instead

Possible problem: Neither dCEF nor CEF support the feature conf

igured.

Page 351: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

351

Solution Step 1 While dCEF supports a large range of

features, some features are not supported. In this case the packets are forwarded to the next slower level of switching. If dCEF cannot process the packet, the router tries to use CEF; if the packet cannot be processed with CEF switched it will be fast switched.

Check the following document to know exactly what is supported:

IOS features supported by CEF and dCEF matrix

Page 352: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

352

Solution(cont.) Step 2 Even if neither dCEF nor CEF ar

e enabled to forward the packet to a destination, the CEF table is built anyway but the adjacency is called a “punt”. See table 4 for further troubleshooting steps.

Page 353: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Packets are not dCEF switched but CEF switched instead.

Possible problem: dCEF does not support the feature configure

d.

Page 354: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

354

Solution Step 1 While dCEF supports a large range of featur

es, However some features or hardware are not supported. In this case the packets are forwarded to the next higher level of switching, which is centralized. In this case the feature configured is supported by CEF.

Check the following documents to know exactly what is supported:

- IOS features supported by CEF and dCEF matrix - IOS hardware support of CEF and dCEF matrix

Page 355: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

355

Solution(cont.) Step 2 Make certain dCEF is configured on the rout

ing using the following commands: show cef interface x/y, show ip cef <prefix>, show adjacency <interface> detail Ultimately issue show cef not-switch and look for

packets that are not dCEF switched. Use the show interface <interface> stats comma

nd to figure out if the packets are dCEF switched.

Page 356: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

356

ExampleRouter# show interface FastEthernet0/0/0 statsFastEthernet0/0/0 Switching path Pkts In Chars In Pkts Out Chars Out Processor 3736 383776 280261 387230212 Route cache 3800 390105 380276 487270282 Distributed cache 0 0 0 0 Total 7536 773881 660537 874500494In this case none of the packets going through the interface are

dCEF switched. All the counters for “Distributed cache” row are still zero.

Page 357: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Load-sharing is not working

Possible problem: The traffic includes only a small

number of flows.

Page 358: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

358

Solution Step1 Check the load-sharing mode. Load-sharing

during CEF/dCEF switching has two modes: Per-destination (original, tunnel, universal) Per-packet. If you do not want to enable per-packet load shari

ng, enable the per-destination tunnel algorithm that is designed to give a good traffic load-sharing when the number of flows is small. Issue the global following command:

ip cef load-sharing algorithm tunnel

Page 359: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Load-sharing is not working

Possible problem: The network topology is subject to

traffic polarization

Page 360: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

360

Solution Step1 Per destination load sharing is based upon a hash function

to perform the balancing. This hash function is subject to polarization, canceling the benefits of load sharing – some routers, under specific traffic distribution patterns, assign all the sessions to the same link regardless of other available paths. The next hop will perform the same hash algorithm and will assign all the traffic to a single link.

Check the load-sharing mode. Load-sharing during CEF/dCEF switching has two modes:

Per-destination (old algorithm, new algorithm, tunnel) Per-packet. Issue the following global command on all the routers in the net

work: ip cef load-sharing algorithm universal This modifies the hash algorithm on each router to correct the eff

ects of polarization. You can also enable per packet load sharing.

Page 361: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Load-sharing is not working

Possible problem: Only one next-hop is accessible to go to

a destination.

Page 362: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

362

SolutionStep1 Make certain there are multiple valid next-hops t

o the destination. If the data streams are destined to different hosts, then we should have load sharing. If not, check to see if all the possible parallel paths are up and their next-hop are reachable. You can issue the following commands:

show ip route <prefix> and look for more than 1 next-hop.

show ip cef <prefix> and look for more that 1 next-hop.show ip cef <prefix> internalshow ip cef exact-route <source address> <destination

address>show adjacency detail on those next hop interfaces.

Page 363: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

The route processor runs out of memory when dCEF is enabled

Possible problem:  CEF messages are queued in

the route processor.

Page 364: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

364

Solution Step1 Check the memory available on the route proces

sor. dCEF is downloading CEF tables to all of the linecards of the router if distributed CEF is turned on. During this process the route processor may queue a lot of CEF messages, and run out of memory.

The following commands can be issued on the route processor to track the problem:

show ip cef summary show memory summary show process memory The depth of the queue of IPC information elements (x

drs) packed into IPC messages sent from the route processor to the line card is reported in the output of the following command:

Page 365: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

365

Examplerouter#show cef linecard detailCEF linecard slot number 0, status up Sequence number 46, Maximum sequence number expected 118, Seq Ep

och 1 Send failed 0, Out Of Sequence 0, drops 0 Linecard CEF reset 0, reloaded 1 1208 elements packed in 944 messages(50328 bytes) sent 0 elements cleared linecard disabled - failed a reload 0/0/0 xdr elements in LowQ/MediumQ/HighQ 24/9/25 peak elements on LowQ/MediumQ/HighQ Input packets 0, bytes 0 Output packets 0, bytes 0, drops 0 CEF Table statistics: Table name Version Prefix-xdr Status Default-table 106 117 Active, sync

Page 366: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

366

Solution(cont.) Step2 Tuning the windowing mechanism u

sed for the route processor/linecard communication will help fixing the problem. Increasing this may reduce the queuing, and thus memory used on the route processor.

With the following command you will change the default value (100 Kbytes) of the window:

ip cef linecard ipc memory <Kbytes>

Page 367: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

CEF disables itself on a linecard due to lack of memory

Possible problem: The linecard has a memory short

age.

Page 368: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

368

Solution Step1 Check the memory available on the linecards. d

CEF downloads CEF tables to all of the linecards of the router if distributed CEF is turned on, even if you aren't CEF switching on that interface. Because of this, all linecards must have the memory necessary to carry all the routes.

The following commands can be issued on each linecards to track the problem:

show ip cef summary show memory summary show process memory show cef linecard On a 7500, enter the VIP card with if-con <x> console.

Page 369: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

CEF Case Study

Page 370: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

370

Case Study(topology)

FDDIDual Ring

IGRP-555 BGP-AS-333

BGP AS-111Local Preference 2000

R1

R3 R2

12.0.0.1

12.0.0.2 12.0.0.3

11.0.0.2

11.0.0.1

13.0.0.3

13.0.0.1

Nets99, 192, 195,197

BGP AS-222Local Preference 100

Page 371: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

371

R1’s Output

R1#show ip routeCodes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default U - per-user static route Gateway of last resort is not set B 2.0.0.0/8 [200/0] via 13.0.0.1, 18:17:13B 197.12.11.0/24 [200/0] via 11.0.0.1, 18:17:50B 99.0.0.0/8 [200/0] via 11.0.0.1, 18:17:50B 197.12.12.0/24 [200/0] via 11.0.0.1, 18:17:51B 195.31.54.0/24 [200/0] via 11.0.0.1, 18:17:51I 11.0.0.0/8 [100/1110] via 12.0.0.2, 00:00:49, Fddi1/1/0B 195.27.191.0/24 [200/0] via 11.0.0.1, 18:17:51B 197.28.99.0/24 [200/0] via 11.0.0.1, 18:17:51C 12.0.0.0/8 is directly connected, Fddi1/1/0B 192.12.1.0/24 [200/0] via 11.0.0.1, 18:17:51B 192.33.1.0/24 [200/0] via 11.0.0.1, 18:17:51B 195.181.32.0/24 [200/0] via 11.0.0.1, 18:17:51I 13.0.0.0/8 [100/252] via 12.0.0.3, 00:00:00, Fddi1/1/0B 197.21.251.0/24 [200/0] via 11.0.0.1, 18:17:51B 197.111.82.0/24 [200/0] via 11.0.0.1, 18:17:51B 192.18.2.0/24 [200/0] via 11.0.0.1, 18:17:51

Page 372: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

372

R1’s Output (cont.)

R1#show ip cefPrefix Next Hop Interface0.0.0.0/32 receive2.0.0.0/8 12.0.0.3 Fddi1/1/011.0.0.0/8 12.0.0.2 Fddi1/1/012.0.0.0/8 attached Fddi1/1/012.0.0.0/32 receive12.0.0.1/32 receive12.0.0.2/32 12.0.0.2 Fddi1/1/012.0.0.3/32 12.0.0.3 Fddi1/1/012.255.255.255/32 receive13.0.0.0/8 12.0.0.3 Fddi1/1/099.0.0.0/8 12.0.0.2 Fddi1/1/0192.12.1.0/24 12.0.0.2 Fddi1/1/0192.18.2.0/24 12.0.0.2 Fddi1/1/0192.33.1.0/24 12.0.0.2 Fddi1/1/0195.27.191.0/24 12.0.0.2 Fddi1/1/0195.31.54.0/24 12.0.0.2 Fddi1/1/0195.181.32.0/24 12.0.0.2 Fddi1/1/0197.12.11.0/24 12.0.0.2 Fddi1/1/0197.12.12.0/24 12.0.0.2 Fddi1/1/0197.21.251.0/24 12.0.0.2 Fddi1/1/0197.28.99.0/24 12.0.0.2 Fddi1/1/0197.111.82.0/24 12.0.0.2 Fddi1/1/0224.0.0.0/4 receive255.255.255.255/32 receive

Page 373: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

373

R1’s Output (cont.)

R1#show adj detailProtocol Interface AddressIP Fddi1/1/0 12.0.0.2(31) 0 packets, 0 bytes 5000000C0B 1C8000603E617C28AAAA030000000800 ARP 01:33:09 IP Fddi1/1/0 12.0.0.3(9) 0 packets, 0 bytes 5000603E28 B22800603E617C28AAAA030000000800 ARP 01:32:43

R1#show ip cef 195.181.32.0 d195.181.32.0/24, version 17, cached adjacency 12.0.0.20 packets, 0 bytes via 11.0.0.1, 0 dependencies, recursive next hop 12.0.0.2, Fddi1/1/0 via 11.0.0.0/8 valid cached adjacency

R1#show ip cef 11.0.0.1 detail11.0.0.0/8, version 9, cached adjacency 12.0.0.20 packets, 0 bytes via 12.0.0.2, Fddi1/1/0, 12 dependencies next hop 12.0.0.2, Fddi1/1/0 valid cached adjacency

Page 374: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

CEF Case Study

TroubleshootingTroubleshooting

Page 375: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

375

Useful Information

Verify CEF/DCEF entryVerify adjacency entryCheck interface switching modeCheck all features that CEF/DCEF supports

Page 376: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

376

Useful Information (cont.)

Use “show” and “debug” commands

show cef dropshow cef not-cef-switchshow cef [linecard] [interface]

Verify with other switching modes

Page 377: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

377

Show Outputscenter#show cef dropCEF Drop StatisticsSlot Encap_fail Unresolved Unsupported No_route No_adj ChksumErrRP 4 0 0 0 0 02 0 0 0 0 0 03 0 0 0 2452 0 04 0 0 0 0 0 05 0 0 0 0 0 0

center#show cef not-cef-switchedCEF Packets passed on to next switching layerSlot No_adj No_encap Unsupp'ted Redirect Receive Bad_ttl Options AccessRP 0 0 41462 0 7922 0 0 02 0 0 0 0 0 0 0 03 0 0 357 0 1161 0 0 04 0 0 0 0 23 3250 0 05 0 0 216504 0 0 0 0 0

Page 378: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

378

center#show cef int f9/0/0Fddi9/0/0 is up (if_number 26) Internet address is 190.1.0.2/16 ICMP redirects are always sent Per packet loadbalancing is disabled Inbound access list is 1 Outbound access list is 3 Hardware idb is Fddi9/0/0 Fast switching type 2, interface type 8 IP Distributed CEF switching enabled Fast flags 0x5. ifindex 25(25) Slot 9 Slot unit 0 VC -1 Hardware transmit queue ptr 0x48001B80 (0x48001B80) Transmit limit accumulator 0x48001B82 (0x48001B82) IP MTU 4352

center#show cef linecard 9CEF linecard slot number 9, status up, sync Linecard CEF version number 246 Sequence number 1364, Maximum sequence number expected 1388 Send failed 0, Out Of Sequence 0 Linecard CEF reset 1, reloaded 1 190/10761/178 prefix/adjacency/interface elements queued 11022 elements packed in 1365 messages(235456 bytes) sent 0/0 xdr elements in LowQ/HighQ Input packets 7050739, bytes 2838181523 Output packets 7042682, bytes 2799387214, drops 0

Show Outputs (cont.)

Page 379: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

379

Show Outputs (cont.)

VIP-Slot5#show ip cef sum IP Distributed CEF with switching (Table Version 223) 223 routes, 0 unresolved routes (0 old, 0 new) 223 leaves, 168 nodes, 199464 bytes, 724 inserts, 501 invalidations 0 load sharing elements, 0 bytes, 0 references 4 CEF resets, 0 revisions of existing leaves refcounts: 36926 leaf, 36706 node VIP-Slot5#show ip cacheIP radix m-way trie cache 0 entries, 0 tables 0 unique encapsulations, 0 total bytes used by cache 7 adds, 7 invalidates, 46024768 switched, 36807 misses misses: 0 frags, 0 runts, 0 chksm, 0 access 0 badiplen Prefix/Length Age Interface MAC Header IP Flow Cache, 0 active, 512 alloced, 0 opened 0 closed: 0 tcp-fin, 0 idle-time, 0 prefix-inval 0 create failures, 0 switched

VIP-Slot5#show VIP memdMEMD statistics : 0 local-switched misses: 0 tqls, 0 bhs, 0 txaccs, 0 deferred

Page 380: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

380

Debug Commands

• debug ip cef [drop] | [receive] | [events] | [prefix-ipc] | [table] | [ipc] | [interface-ipc]

• debug ip [packet] [error]

• debug adjacency

Page 381: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

381

center#debug ip cef table

Debug Outputs

Nov 13 12:31:21.075: CEF-IP: Receive address 99.0.0.0/32 already existsNov 13 12:31:21.075: CEF-IP: Receive address 99.255.255.255/32 already existsNov 13 12:31:21.075: CEF-Table: Event up, 2.0.0.0/8Nov 13 12:31:21.075: CEF-Table: Event up, 197.12.11.0/24Nov 13 12:31:21.087: CEF-Table: Event up, 197.21.251.0/24Nov 13 12:31:21.087: CEF-Table: Event up, 192.18.2.0/24Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 4 Seq 0 Nov 13 12:31:36.067: CEF-Table: attempting to resolve 197.12.12.0/24Nov 13 12:31:36.067: CEF-IP: resolved 197.12.12.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1Nov 13 12:31:36.067: CEF-Table: attempting to resolve 197.12.11.0/24Nov 13 12:31:36.067: CEF-IP: resolved 197.12.11.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1Nov 13 12:31:57.695: CEF-IP: Checking dependencies of 12.0.0.0/8Nov 13 12:31:57.695: CEF-Table: Adjacency-prefix 12.0.0.1/32 add request -- succeeded Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.27.191.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.31.54.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 195.181.32.0/24Nov 13 12:33:03.079: CEF-Table: Flushing entry for 197.12.11.0/24

Page 382: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

382

Debug Output (cont.)

VIP-Slot9#debug ip packIP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0IP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0

VIP-Slot9#debug ip cef dropIP-CEF: No route found for 34.1.1.1IP-CEF: No route found for 34.1.1.1

border2#debug adjacency ADJ: add 16.2.0.2 (Ethernet0/0) via ARP for 01:21:15ADJ: add 16.2.0.1 (Ethernet0/0) via ARP for 01:20:37ADJ: add 62.0.0.1 (Ethernet0/1) via ARP for 01:20:32ADJ: add 66.0.0.1 (Ethernet0/2) via ARP for 01:22:13ADJ: add 161.2.1.50 (Fddi1/0) via ARP for 03:45:59ADJ: add 180.2.0.2 (FastEthernet8/0/0) via ARP for 01:22:13ADJ: add 200.2.0.2 (Fddi10/0/0) via ARP for 01:23:07ADJ: add 16.2.146.97 (Ethernet0/0) via ARP for 01:23:07ADJ: add 0.0.0.0 (ATM 5/0/0.100) via ATM-PVC for 00:00:00ADJ: add 0.0.0.0 (ATM 5/0/0.103) via ATM-PVC for 00:00:00ADJ: add 0.0.0.0 (Serial4/0/0) via FIB for 00:02:59ADJ: add 0.0.0.0 (Serial4/0/1) via FIB for 00:03:00

Page 383: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Troubleshooting IP Multicast

0981_03F8_c3NW98_US_112

Page 384: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

384

Troubleshooting ToolsTroubleshooting Tools Basic Troubleshooting Advanced

Troubleshooting Case studies

Agenda

Page 385: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

385

“show ip igmp group” command “show ip igmp interface” command “show ip pim neighbor” command “show ip pim interface” command “show ip rpf ” command “show ip mroute” commands

Troubleshooter’s “Hand” Tools

Page 386: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

386

Troubleshooter’s “Hand” Tools

“show ip pim rp” command “show ip pim rp map” comman

d

Special Sparse Mode Tools

Page 387: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

387

show ip igmp group

Shows: Currently joined multicast groups.

Troubleshooting usage: Verify that a receiver has actually joined

the target group If not, use “show ip igmp interface” to check for

proper igmp version, querier, timers, etc. Use “debug ip igmp” to verify that proper igmp

host-router exchange is happening Watch for IGMP v1-v2 interoperability problems

Page 388: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

388

show ip igmp group

R4#show ip igmp groupsIGMP Connected Group MembershipGroup Address Interface Uptime Expires Last Reporter224.1.1.1 Ethernet1 3d16h 00:01:59 172.16.7.2224.0.1.40 Ethernet0 4d15h never 172.16.6.2

Page 389: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

389

show ip igmp interface

Shows: Key IGMP timers, status, etc.

Troubleshooting usage: Verify that correct IGMP version is running Verify that timers are set properly Verify that correct router is IGMP Querier

If not, use “debug ip igmp” to determine what’s wrong

Page 390: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

390

show ip igmp interface

R4#show ip igmp interfaceEthernet1 is up, line protocol is up Internet address is 172.16.7.1, subnet mask is 255.255.255.0 IGMP is enabled on interface Current IGMP version is 2 CGMP is disabled on interface IGMP query interval is 60 seconds IGMP querier timeout is 120 seconds IGMP max query response time is 10 seconds Inbound IGMP access group is not set Multicast routing is enabled on interface Multicast TTL threshold is 0 Multicast designated router (DR) is 172.16.7.1 (this system) IGMP querying router is 172.16.7.1 (this system) No multicast groups joined

Page 391: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

391

show ip pim neighbor

Shows: PIM Neighbor Adjacencies

Troubleshooting usage: Verify that all neighbors are up and using

proper mode If not, check router configs and/or interface status Use “debug ip pim” to observe PIM Query msg

exchange

Page 392: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

392

show ip pim neighbor

R6#show ip pim neighborPIM Neighbor TableNeighbor Address Interface Uptime Expires Mode172.16.10.2 Serial0 4d15h 00:01:19 Dense172.16.11.2 Serial1 4d15h 00:01:00 Dense172.16.9.1 Ethernet0 4d15h 00:01:00 Dense

Page 393: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

393

show ip pim interface

Shows: PIM Interface information.

Mode, Neighbor Count, DR

Troubleshooting usage: Verify correct PIM mode is configured

on interface(s) If not, check router configs

Verify Designated Router is correct If not, check router configs Especially critical for Sparse Mode!

Page 394: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

394

show ip pim interface

R6#show ip pim interfaceAddress Interface Mode Nbr Query DR Count Intvl172.16.10.1 Serial0 Dense 1 30 0.0.0.0172.16.11.1 Serial1 Dense 1 30 0.0.0.0172.16.9.2 Ethernet0 Dense 1 30 172.16.9.2

Page 395: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

395

show ip rpf

Shows: RPF interface information for source

Troubleshooting usage: Verify that RPF information is correct

If not, check unicast routing data for correctness Ping or Trace “source” to verify unicast route is

working. (Fix any unicast routing problems first!) May need to use DVMRP routes or Static Mroutes

to fix unicast-multicast incongruency

Page 396: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

396

show ip rpf

R4#show ip rpf 172.16.8.1RPF information for Source1 (172.16.8.1) RPF interface: Ethernet0 RPF neighbor: R3 (172.16.6.1) RPF route/mask: 172.16.8.0/255.255.255.0 RPF type: unicast

R4#sh ip rpf 172.16.12.2RPF information for Source2 (172.16.12.2) RPF interface: Tunnel0 RPF neighbor: R6 (172.16.11.1) RPF route/mask: 172.16.12.0/255.255.255.0 RPF type: DVMRP

Page 397: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

397

show ip mroute commands

“show ip mroute sum” “show ip mroute count” “show ip mroute active” “show ip mroute”

Page 398: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

398

show ip mroute summary

Shows: Multicast state at a glance

Active groups Active senders in the group. (If SPT joined)

Troubleshooting usage: Verify multicast group(s) are active.

If not, check for group state at RP. (Sparse mode) Work your way from a known source to a receiver

or the RP to find where things stop Verify senders are active. (If SPT joined)

If not, check state in 1st-hop router Verify sender is really sending

Page 399: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

399

show ip mroute summary

dallas-gw>show ip mroute summaryIP Multicast Routing TableFlags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPTTimers: Uptime/ExpiresInterface state: Interface, Next-Hop, State/Mode

(*, 224.0.255.255), 6d23h/00:02:59, RP 171.69.10.13, flags: SJC (171.68.37.121/32, 224.0.255.255), 6d23h/00:02:55, flags: CT (171.69.58.88/32, 224.0.255.255), 6d23h/00:02:58, flags: CT (171.69.60.189/32, 224.0.255.255), 6d23h/00:02:55, flags: CT (171.69.128.115/32, 224.0.255.255), 3d01h/00:02:58, flags: CJT (171.69.199.49/32, 224.0.255.255), 6d23h/00:02:57, flags: CT (171.70.247.82/32, 224.0.255.255), 01:55:33/00:02:58, flags: CJT...

Page 400: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

400

show ip mroute count

Shows: Multicast traffic flow rates, drops, etc.

Group traffic summary Sender rates, packet counts, drops, etc.

Troubleshooting usage: Verify multicast traffic is being received

If not, work your way from source to receiver tofind where things stop

Verify multicast traffic is being forwarded If not, why? “oif null”, “rpf-failure”

Page 401: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

401

show ip mroute countdallas-gw>show ip mroute 224.0.255.255 countIP Multicast Statistics - Group count: 7, Average sources per group: 3.28Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per secondOther counts: Total received/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 224.0.255.255, Source count: 6, Group pkt count: 538169 RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0 Source: 171.68.37.121/32, Forwarding: 120484/0/94/0, Other: 120535/0/51 Source: 171.69.58.88/32, Forwarding: 120281/0/84/0, Other: 120283/2/0 Source: 171.69.60.189/32, Forwarding: 120445/0/95/0, Other: 120447/2/0 Source: 171.69.128.115/32, Forwarding: 53018/0/93/0, Other: 53018/0/0 Source: 171.69.199.49/32, Forwarding: 120414/1/92/0, Other: 120415/1/0 Source: 171.70.247.82/32, Forwarding: 3527/1/78/0, Other: 3527/0/0

Page 402: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

402

show ip mroute active

Shows: Sources with traffic rates above threshold

Aggregate RP Tree and (S, G) rates shown Rates in Kbps (1 sec, 1 min, 5 min avgs.)

Troubleshooting usage: Determine which sources/groups are active Determine the traffic rate of each source

Note: Must have switched to Shortest-Path tree Verify “target” group multicast traffic is

being received If not, work your way from source to receiver

Page 403: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

403

show ip mroute activebarrnet-gw>show ip mroute activeActive IP Multicast Sources - sending >= 4 kbps

Group: 224.2.156.43, *cisco: Bloomington IPTV Beacon Source: 172.17.67.43 (bloom-iptv.cisco.com) Rate: 6 pps/63 kbps(1sec), 65 kbps(last 19 secs), 37 kbps(life avg)

Group: 224.2.154.118, Radio Bandit Source: 192.36.125.68 (falcon.pilsnet.sunet.se) Rate: 11 pps/30 kbps(1sec), 30 kbps(last 33 secs), 23 kbps(life avg) Group: 224.2.246.13, UO Presents KWAX Classical Radio Source: 128.223.83.204 (d83-204.uoregon.edu) Rate: 24 pps/69 kbps(1sec), 72 kbps(last 2 secs), 70 kbps(life avg) Group: 224.2.180.115, ANL TelePresence Microscopy Site Source: 146.139.72.5 (aem005.amc.anl.gov) Rate: 1 pps/5 kbps(1sec), 9 kbps(last 52 secs), 12 kbps(life avg)...

Page 404: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

404

show ip mroute

Shows: Detailed multicast state in the router

Troubleshooting usage: Verify Incoming Interface is correct

If not, check unicast routing table (May need touse DVMRP routes or Static mroutes)

Verify Outgoing Interface(s) are correct If interface incorrectly “Pruned”, check state in

downstream router? May need to “debug ip pim <group>” to

determine problem

Page 405: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

405

show ip mroutebarrnet-gw>show ip mrouteIP Multicast Routing TableFlags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPTTimers: Uptime/ExpiresInterface state: Interface, Next-Hop, State/Mode(*, 224.2.130.100), 00:18:53/00:02:59, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Fddi1/0, Forward/Dense, 00:09:20/00:02:38 Hssi3/0, Forward/Dense, 00:18:53/00:00:00(208.197.169.209/32, 224.2.130.100), 00:18:53/00:02:27, flags: T Incoming interface: Hssi3/0, RPF nbr 131.119.26.9 Outgoing interface list: Fddi1/0, Forward/Dense, 00:16:16/00:02:38(*, 239.100.111.224), 05:35:08/00:02:58, RP 171.69.10.13, flags: DP Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Null

Page 406: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

406

show ip pim rp map

Shows: RP assignments by multicast group range

Troubleshooting usage: Verify that configured (static or Auto-RP)

RP’s are correct If not, check local router config

and/or network Auto-RP configuration

Page 407: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

407

show ip pim rp map

dallas-gw>show ip pim rp mapPIM Group-to-RP Mappings

Group(s) 224.0.0.0/4, uptime: 6d21h, expires: 00:02:56 RP 171.69.10.13 (sj-eng-mbone.cisco.com) Info source: 192.31.7.37 (barrnet-gw.cisco.com)

Page 408: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

408

show ip pim rp

Shows: RP’s by active group

Troubleshooting usage: Verify that the RP for the target multicast

group is correct If not, check RP mapping, local router RP conf

ig and/or network Auto-RP

Page 409: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

409

show ip pim rp

dallas-gw>sh ip pim rpGroup: 224.2.127.253, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.1.127.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.2.127.254, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.0.255.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34Group: 224.2.0.1, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34

Page 410: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

410

Troubleshooter’s “Power” Tools

“mtrace” and “mstat” commands

“mrinfo” command “show ip mpacket” command

Page 411: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

411

mtrace and mstat commands

Based on Unix “mtrace” command Split into two separate commands Both use the same mechanism

draft-ietf-idmr-traceroute-ipm-xx.txt

Page 412: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

412

mtrace

Shows: Multicast path from source to receiver

Similar to unicast “trace” command Trace path between any two points in network TTL Thresholds and Delay shown at each node

Troubleshooting usage: Find where multicast traffic flow stops

Focus on router where flow stops Verify path multicast traffic is following

Identify sub-optimal paths

Page 413: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

413

mstat

Shows: Multicast path in pseudo graphic format

Trace path between any two points in network Drops/Duplicates shown at each node TTLs and Delay shown at each node

Troubleshooting usage: Locate congestion point in the flow

Focus on router with high drop/duplicate count Duplicates indicated as “negative” drops

Page 414: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

414

Multicast Dist. Tree

Mtrace Packet

src dest

mtrace re

quest

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

mtrace response

Unix Workstationor

Cisco Router

Last-hopRouter

First-hopRouter

Mtrace Packet Flow

mtrace/mstat—How it Works

Note: Mtrace packets use specialIGMP packets with IGMP Type codes of 0x1E and 0x1F.

Page 415: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

415

mtrace/mstat—How it Works

Each hop adds data to packet Query arrival time Incoming Interface Outgoing Interface Prev. Hop Router address Input packet count Output packet count Total packets for this Source/Group Routing Protocol TTL Threshold Fowarding/Error Code

Page 416: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

416

mtrace

dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>

Page 417: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

417

mstatdallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source

350981_03F8_c3NW98_US_112

Page 418: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

418

mstatdallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source

360981_03F8_c3NW98_US_112

Page 419: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

419

mrinfo

Shows: Multicast neighbor router information

Indicates router’s capabilities and code version Multicast interface information

TTL-Thresholds, Metric, Protocol, Status

Troubleshooting usage: Verify multicast neighbors.

Confirm bi-directional neighbor adjacency exists Verify Tunnels are up in both directions

Page 420: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

420

mrinfodallas-gw>mrinfo paloalto-mbone1.bbnplanet.netTranslating " paloalto-mbone1.bbnplanet.net "...domain server (171.68.10.70) [OK]131.119.0.197 (paloalto-mbone1.bbnplanet.net) [version cisco 11.2] [flags: PMSA]: 131.119.0.197 -> 131.119.0.201 (paloalto-cr1.bbnplanet.net) [1/0/pim] 131.119.244.244 -> 0.0.0.0 [1/32/pim/querier] 131.119.0.197 -> 204.162.119.8 (hydra.precept.com) [1/32/tunnel/querier] 192.42.110.249 -> 192.9.9.71 (mbone.Sun.COM) [1/32/tunnel] 192.42.110.249 -> 204.123.13.69 (chocolate.research.digital.com) [1/32/tunnel] 192.42.110.249 -> 36.253.0.11 (alpo.Stanford.EDU) [1/32/tunnel] 131.119.0.197 -> 0.0.0.0 [1/64/tunnel/pim/querier/leaf] 131.119.0.197 -> 0.0.0.0 [1/32/tunnel/pim/querier/leaf] 192.42.110.249 -> 204.94.211.39 (sgi-too.SGI.COM) [4/64/tunnel/querier] 192.42.110.249 -> 192.216.174.1 [1/32/tunnel/querier/down/leaf] 192.42.110.249 -> 198.94.216.2 [1/32/tunnel/querier/down/leaf] 192.42.110.249 -> 204.161.60.33 (berkeley.faslab.com) [1/32/tunnel/querier] 131.119.0.197 -> 204.154.181.12 [1/32/tunnel/querier/down/leaf]...

Page 421: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

421

show ip mpacket

Used to view multicast packet headers Command syntax

show ip mpacket <source> <group> [detail] You can view:

{source, group} traffic pairs IP ident and ttl Inter-packet delay

Configure multicast header capture first “ip multicast cache-headers” config cmd Captures multicast headers in 1024 entry ring buffer

Page 422: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

422

show ip mpacketdino-cisco-fr#show ip mpacket 224.2.231.173IP Multicast Header Cache - entry count: 29, next index: 30Key: id/ttl timestamp (name) source group

D782/117 206416.908 (all-purpose-gunk.near.net) 199.94.220.184 224.2.231.1737302/113 206417.172 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1736CB2/114 206417.412 (wayback.uoregon.edu) 128.223.156.117 224.2.231.173D786/117 206417.868 (all-purpose-gunk.near.net) 199.94.220.184 224.2.231.173E2E9/123 206418.488 (dino-ss20.cisco.com) 171.69.58.81 224.2.231.1731CA7/127 206418.544 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAA/127 206418.584 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAC/127 206418.624 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CAF/127 206418.664 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CB0/127 206418.704 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1731CB2/127 206418.744 (dino-ss2.cisco.com) 171.69.129.220 224.2.231.1732BBB/114 206418.840 (crevenia.parc.xerox.com) 13.2.116.11 224.2.231.1733D1D/123 206419.380 (dalvarez-ss20.cisco.com) 171.69.60.189 224.2.231.1732BC0/114 206419.672 (crevenia.parc.xerox.com) 13.2.116.11 224.2.231.1737303/113 206419.888 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1737304/113 206420.140 (speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.1732C7E/123 206420.360 (lwei-ss20.cisco.com) 171.69.58.88 224.2.231.173

Page 423: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

423

Troubleshooting Tools Basic TroubleshootingBasic Troubleshooting Advanced

Troubleshooting Case studies

Agenda

Page 424: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

424

Basic Troubleshooting

Is each piece working correctly?

Troubleshooting Table

SourceSource NetworkNetwork ReceiversReceivers

StateState

Packet Flow Packet Flow ?? ?? ??

?? ??NANA

Page 425: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

425

Check Source Packet Flow

Check interface counters on source Check source TTL > 1

Verify TTL setting in application Confirm on upstream router

show ip traffic Increasing “Invalid hop count”

Page 426: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

426

Check Source Packet Flow

Check 1st-Hop router for traffic flow show ip mroute count show ip mroute active show ip mpacket

Don’t forget to turn on: ip multicast cache-headers

debug ip mpacket Use with caution!! “detail” or ACL for granularity

Page 427: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

427

Check Network State

Most complex piece Depends of protocol, mode, etc. Check initial state creation Check for pruning and timer

expiration during session

Page 428: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

428

Network State

show/debug ip mroute commands watch oilist for null entries

show/debug ip pim commands show/debug ip dvmrp commands show ip rpf mtrace command

Page 429: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

429

PIM SM Troubleshooting

show ip pim rp [<group>] indicates RP for the group

show ip pim rp mapping indicates RP for the group

debug ip pim auto-rp

Page 430: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

430

Check Network Packet Flow

show ip mroute count show ip mroute active show ip mpacket

Turn on “ip multicast cache-headers” first

debug ip mpacket Be Careful with this one!

mstat

Page 431: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

431

Check Receiver State

show ip igmp interface show ip igmp group debug ip igmp IGMPv1 vs. IGMPv2

Page 432: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

432

Check Receiver Packet Flow

Check receiver interface stats Is the stack installed and configured

properly? Is the application installed and

configured properly? Watch for duplicates

Performance implication

Page 433: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

433

Troubleshooting Tools Basic Troubleshooting Advanced Advanced

TroubleshootingTroubleshooting Case studies

Agenda

Page 434: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

434

Advanced Troubleshooting

Troubleshooting Network Troubleshooting Network StateState

Troubleshooting PIM-DVMRP Troubleshooting ATM P2MP

VCs

Page 435: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

435

Troubleshooting Network State

Use “show ip mroute <group>” Specify target group to limit output. DM: Trace state from source to receiver. SM: Trace state from receiver to RP then from source

to RP Use “debug ip pim <group>”

Be sure to specify target group to limit debug output! Use with caution!!

Use “debug ip mroute <group>” Be sure to specify target group to limit debug output! Use with caution!!

Page 436: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

436

Mroute Flags

“S”—Sparse Mode Appears only on the (*, G) entries

“D”—Dense Mode Appears only on the (*, G) entries

“P”—Pruned Sparse mode: oilist is null Dense mode: all interfaces in oilist = Pruned

Page 437: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

437

Mroute Flags (Cont.)

“C”—Connected A rcvr for this group is directly connected

to this router. “L”—Local

The router itself is a member of this group and is receiving group traffic.

“T”—Shortest-path Tree (SPT) flag Set on (S,G) entry when packets are being

successfully received via the SPT

Page 438: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

438

Mroute Flags (Sparse Mode Only)

“R”—RP bit Only appears on (S, G) entries (S, G) state is associated with Shared Tree RPF interface points up Shared Tree to RP Used to prune unwanted (S, G) traffic from

the Shared Tree after SPT switchover “F”—Register Flag

Appears on (S, G) entries Set on (*, G) if any (S, G) “F” flags set Router is directly connected to a source Register messages must be sent to RP

Page 439: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

439

Mroute Flags (Sparse Mode Only)

“J” —Join SPT Set once/sec when SPT-Threshold exceeded Switch to SPT for next (S, G) packet rcvd’d

(*, G) Entry

“J” —SPT Joined SPT Joined due to SPT-Threshold exceeded. Switch back to Shared Tree if traffic rate falls below SPT-Threshold. (Checked once/min)

(S, G) Entry

Page 440: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

440

Network Mroute State Examples

PIM DM PIM SM

Joining Registering SPT-Switchover

Page 441: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

441

PIM DM

11 “rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”.

11 Multicast Packets(128.9.160.43, 224.2.127.254)

S0

rtr-a

rtr-b

S1

E1

S0

S3

Page 442: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

442

(*, 224.2.127.254), 00:00:10/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial1, Forward/Dense, 00:00:10/00:00:00 Serial3, Forward/Dense, 00:00:10/00:00:00

(128.9.160.43/32, 224.2.127.254), 00:00:10/00:02:49, flags: T Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Serial1, Forward/Dense, 00:00:10/00:00:00 Serial3, Forward/Dense, 00:00:10/00:00:00

Initial “Flooding” Statein “rtr-a”

PIM DMS0

rtr-a

rtr-b

Multicast Packets(128.9.160.43, 224.2.127.254)

S1

E1

S0

S3

Page 443: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

443

PIM DM

11 “rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”

Multicast Packets(128.9.160.43, 224.2.127.254)

22 “rtr-b” is a leaf node w/o receivers. Sends Prune for (S,G)

Prune22

33 “rtr-a” Prunes interface for (S,G)

S0

rtr-a

rtr-b

S1

E1

S0

33

XX

S3

Page 444: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

444

(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial1, Forward/Dense, 00:00:12/00:00:00 Serial3, Forward/Dense, 00:00:12/00:00:00

(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: T Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Serial1, Forward/Dense, 00:00:12/00:00:00 Serial3, Prune/Dense, 00:00:04/00:02:56

PIM DM

Multicast Packets(128.9.160.43, 224.2.127.254)

S0

rtr-a

rtr-b

S1

E1

S0

S3

State in “rtr-a”after Pruning

Page 445: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

445

(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: Serial0, Forward/Dense, 00:00:12/00:00:00

(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: PT Incoming interface: Serial0, RPF nbr 198.92.1.129 Outgoing interface list: Null

PIM DM

Multicast Packets(128.9.160.43, 224.2.127.254)

S0

rtr-a

rtr-b

S1

E1

S0

S3

State in “rtr-b”after Pruning

Page 446: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

446

• “Rcvr A” wishes to receive group G traffic. Sends IGMP Join for G11

IGMP Join11

• “rtr-b” creates (*,G) state; sends (*,G) PIM Join towards RP22

PIM Join22

• “rtr-a” creates (*,G) state; sends (*,G) PIM Join towards RP33

PIM Join33

Shared Tree44

• Shared tree is built all the way back to the RP44

PIM SM Joining

E0S0 rtr-a

rtr-b

S1

E0E1

Rcvr A

Shared Tree

To RP (10.1.5.1)

10.1.2.2

10.1.2.110.1.4.2

Page 447: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

447

PIM SM Joining

rtr-b>sh ip mroute

(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.2.1 Outgoing interface list: Ethernet1, Forward/Sparse, 00:00:05/00:02:52

State in “rtr-b” after Joining (*, 224.1.1.1)

E0S0 rtr-a

rtr-b

S1

E0E1

To RP (10.1.5.1)

Rcvr A

10.1.2.2

10.1.2.110.1.4.2

Shared Tree

Page 448: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

448

rtr-a>sh ip mroute

(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1 Outgoing interface list: Ethernet0, Forward/Sparse, 00:00:05/00:02:54

State in “rtr-a” after Joining (*, 224.1.1.1)

PIM SM Joining

E0S0 rtr-a

rtr-b

S1

E0E1

To RP (10.1.5.1)

Rcvr A

10.1.2.2

10.1.2.110.1.4.2

Shared Tree

Page 449: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

449

rtr-c>sh ip mroute 224.1.1.1

(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: S Incoming interface: Null, RPF nbr 0.0.0.0, Outgoing interface list: Serial0, Forward/Sparse, 00:03:14/00:02:59 Serial1, Forward/Sparse, 00:03:14/00:02:59

State in “RP” before Registering(with receivers on Shared Tree)

PIM SM Registering

rtr-a

RP

rtr-crtr-b

Shared Tree

S3

S0 S1

171.68.28.139

Page 450: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

450

rtr-a>sh ip mroute 224.1.1.1

No such group.

State in “rtr-a” before Registering(with receivers on Shared Tree)

rtr-a

RP

rtr-crtr-b

Shared Tree

PIM SM Registering

E0

S0

Page 451: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

451

Source 171.68.37.121

• “Source” begins sending group G traffic11

11

(171.68.37.121, 224.1.1.1) Mcast Packets

• “rtr-a” encapsulates packets in Registers; unicasts to RP22

Register Msgs22

• “rtr-c” (RP) de-encapsulates packets; forwards down Shared tree33

33 (*, 224.1.1.1)Mcast Traffic

rtr-a

RP

rtr-crtr-b

Shared Tree

PIM SM Registering

Page 452: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

452

rtr-a>sh ip mroute 224.1.1.1

(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: SP Incoming interface: Serial0, RPF nbr 171.68.28.191, Outgoing interface list: Null

(171.68.37.121/32, 224.1.1.1), 00:00:03/00:02:56, flags: FPT Incoming interface: Ethernet0, RPF nbr 0.0.0.0, Registering Outgoing interface list: Null

State in “rtr-a” while Registering

(171.68.37.121, 224.1.1.1) Mcast Packets

Register Msgs

Source 171.68.37.121

rtr-a

RP

Shared Tree

rtr-crtr-b

(*, 224.1.1.1)Mcast Traffic

PIM SM Registering

E0

S0

70

Page 453: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

453

rtr-a

• RP sends (S,G) Join toward Source; builds SPT44

Join 44Join rtr-b

• “Source” begins sending group G traffic 11

• “rtr-a” encapsulates packets in Registers; unicasts to RP22

• RP (“rtr-c”) de-encapsulates packets; forwards down Shared tree33

RP

rtr-c

Shared Tree

(*, 224.1.1.1)Mcast Traffic

(171.68.37.121, 224.1.1.1) Mcast Packets

Register Msgs

Source 171.68.37.121

PIM SM Registering

Page 454: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

454

• RP begins receiving (S,G) traffic down SPT 55

55

• RP sends “Register-Stop” to “rtr-a”66

Register-Stop66

rtr-a

RP

rtr-crtr-b

Shared Tree

(*, 224.1.1.1)Mcast Traffic

Source 171.68.37.121

(171.68.37.121, 224.1.1.1) Mcast Packets

Register Msgs

PIM SM Registering

Page 455: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

455

• “rtr-a” stops encapsulating traffic in Register Messages77

• (S,G) Traffic now flowing down a single path (SPT) to RP88

88

rtr-a

RP

rtr-crtr-b

Shared Tree

(*, 224.1.1.1)Mcast Traffic

Source 171.68.37.121

(171.68.37.121, 224.1.1.1) Mcast Packets

• RP begins receiving (S,G) traffic down SPT55

• RP sends “Register-Stop” to “rtr-a”66

PIM SM Registering

Page 456: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

456

rtr-a>sh ip mroute 224.1.1.1

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP Incoming interface: Serial0, RPF nbr 171.68.28.191, Outgoing interface list: Null

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: FT Incoming interface: Ethernet0, RPF nbr 0.0.0.0 Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

State in “rtr-a” after Registering

rtr-a

RP

rtr-crtr-b

Shared Tree

(*, 224.1.1.1)Mcast Traffic

Source 171.68.37.121

(171.68.37.121, 224.1.1.1) Mcast Packets

PIM SM Registering

E0

S0

Page 457: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

457

rtr-b>sh ip mroute 224.1.1.1

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Null

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32

State in “rtr-b” after “rtr-a” Registers(with receivers on Shared Tree)

rtr-a

RP

rtr-crtr-b

Shared Tree

(*, 224.1.1.1)Mcast Traffic

Source 171.68.37.121

(171.68.37.121, 224.1.1.1) Mcast Packets

PIM SM Registering

S0 S1

171.68.28.190

Page 458: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

458

rtr-c>sh ip mroute 224.1.1.1

(*, 224.1.1.1), 00:09:21/00:02:38, RP 171.68.28.140, flags: S Incoming interface: Null, RPF nbr 0.0.0.0, Outgoing interface list: Serial0, Forward/Sparse, 00:09:21/00:02:38 Serial1, Forward/Sparse, 00:03:14/00:02:46

(171.68.37.121, 224.1.1.1, 00:01:15/00:02:46, flags: T Incoming interface: Serial3, RPF nbr 171.68.28.139, Outgoing interface list: Serial0, Forward/Sparse, 00:00:49/00:02:11 Serial1, Forward/Sparse, 00:00:49/00:02:11

State in “RP” after “rtr-a” Registers(with receivers on Shared Tree)

rtr-a

RP

rtr-crtr-b

Shared Tree

(*, 224.1.1.1)Mcast Traffic

Source 171.68.37.121

(171.68.37.121, 224.1.1.1) Mcast Packets

PIM SM Registering

S3

S0 S1

171.68.28.139

Page 459: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

459

PIM SM SPT-Switchover Review

Once each second Compute new (*, G) traffic rate If threshold exceeded, set “J” flag in (*,

G) For each (Si , G) packet received:

If “J” flag set in (*, G) Clear “J” flag in (*,G) Join SPT for (Si , G) Mark (Si , G) entry with “J” flag

SPT-Switchover Mechanism

Page 460: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

460

PIM SM SPT-Switchover

Rcvr A

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.5.1, Outgoing interface list: Serial1, Forward/Sparse, 00:01:43/00:02:11

State in “rtr-c” before switch

Rcvr A

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 461: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

461

Rcvr A

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1, Outgoing interface list: Ethernet0, Forward/Sparse, 00:01:43/00:02:11

State in “rtr-a” before switch

PIM SM SPT-Switchover

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 462: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

462

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11

State in “rtr-b” before switch

PIM SM SPT-Switchover

Rcvr A

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 463: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

463

PIM SM SPT-Switchover

E0S0 rtr-a

S1

E010.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

Group “G” rate exceeds SPT Threshold at “rtr-b”;11

Group “G” rate > Threshold11

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11

Set J Flag in (*, G) and wait for next (Si,G) packet22

22

J

rtr-bE1

Rcvr A

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 464: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

464

PIM SM SPT-Switchover

(Si,G) packet arrives down Shared tree.33

E0S0 rtr-a

S1

E010.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SCJ Incoming interface: Ethernet0, RPF nbr 10.1.4.2, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11

33

Clear J Flag in (*,G) and44 55 Send (Si,G) Join towards Si .

44

rtr-bE1

Rcvr A

Group “G” rate exceeds SPT Threshold at “rtr-b”;11

Set J Flag in (*, G) and wait for next (Si,G) packet.22

SC

(Si,G) Join55

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

820981_03F8_c3NW98_US_112

Page 465: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

465

PIM SM SPT-Switchover

Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT66

(Si,G)RP-bit Prune 66

E0S0 rtr-a

rtr-b

S1

E0E1

Rcvr A

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si77

(Si,G) Join77

“rtr-a” forwards (Si,G)RP-bit Prune toward RP88

(Si,G)RP-bit

Prune

88

(Si, G) traffic begins flowing down SPT tree99

(Si,G) Traffic99

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)rtr-c

10.1.4.1

S1S0

Page 466: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

466

PIM SM SPT-Switchover

E0S0 rtr-a

rtr-b

S1

E0E1

Rcvr A

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT66

SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si77

“rtr-a” forwards (Si,G)RP-bit Prune toward RP88

(Si, G) traffic begins flowing down SPT tree99

• (Si, G) traffic ceases flowing down Shared tree1010

1010To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 467: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

467

Rcvr A

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.4.1, Outgoing interface list: Ethernet0, Forward/Sparse, 00:01:43/00:02:11

(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: T Incoming interface: Serial1, RPF nbr 10.1.9.2 Outgoing interface list: Ethernet0, Forward/Sparse, 00:13:25/00:02:30

State in “rtr-a” after switch

Rcvr A

PIM SM SPT-Switchover

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 468: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

468

PIM SM SPT-Switchover

Rcvr A

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC Incoming interface: Ethernet0, RPF nbr 10.1.2.1, Outgoing interface list: Ethernet1, Forward/Sparse, 00:01:43/00:02:11

(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: SCJT Incoming interface: Ethernet0, RPF nbr 10.1.2.1 Outgoing interface list: Ethernet1, Forward/Sparse, 00:13:28/00:02:53

State in “rtr-b” after switch

Rcvr A

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

J Flag indicates(S, G) created by

exceeding theSPT-threshold

SCJT

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 469: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

469

PIM SM SPT-Switchover

Rcvr A

(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S Incoming interface: Serial0, RPF nbr 10.1.5.1, Outgoing interface list: Serial1, Forward/Sparse, 00:01:43/00:02:11

(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: PR Incoming interface: Serial0, RPF nbr 10.1.5.1 Outgoing interface list: Null

State in “rtr-c” after switch

Rcvr A

E0S0 rtr-a

rtr-b

S1

E0E1

10.1.2.2

10.1.2.110.1.4.2

To Source “Si”

(Si, G) Traffic FlowShared (RPT) Tree

SPT Tree

To RP (10.1.5.1)

10.1.4.1

S1S0

rtr-c

Page 470: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

470

Advanced Troubleshooting

Troubleshooting Network State

Troubleshooting PIM-DVMRPTroubleshooting PIM-DVMRP Troubleshooting ATM P2MP

VCs

Page 471: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

471

PIM-DVMRP Troubleshooting

pim-dvmrp-gw:

interface tunnel0ip unnumbered ethernet0ip pim dense-modetunnel mode dvmrptunnel source ethernet0tunnel destination 135.1.22.98

interface ethernet0ip addr 135.1.3.102 255.255.255.0ip pim dense-mode

interface ethernet1ip addr 135.1.2.102 255.255.255.0ip pim dense-mode

Example Network

Site

ISP

pim-dvmrp-gw

Tunnel0

Ethernet0

mrouted

Ethernet1

135.1.2.100

Page 472: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

472

PIM-DVMRP Troubleshooting

Verifying the DVMRP tunnelVerifying the DVMRP tunnel Verifying DVMRP route exchange

Page 473: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

473

Verifying the DVMRP Tunnel

Using the “show interface” Commandpim-dvmrp-gw> show int tunnel 0Tunnel0 is up, line protocol is up Hardware is Tunnel Interface is unnumbered. Using address of Ethernet0 (135.1.3.102) MTU 1500 bytes, BW 9 Kbit, DLY 500000 usec, rely 255/255, load 1/255 Encapsulation TUNNEL, loopback not set, keepalive set (10 sec) Tunnel source 135.1.3.102 (Ethernet0), destination 135.1.22.98 Tunnel protocol/transport IP/IP (DVMRP), key disabled, sequencing disabled Checksumming of packets disabled, fast tunneling enabled Last input 00:00:05, output 00:00:08, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0 (size/max/drops); Total output drops: 0 .

.

.

Page 474: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

474

Verifying the DVMRP Tunnel

pim-dvmrp-gw>mrinfo135.1.3.102 [version cisco 11.2] [flags: PMA]: 135.1.3.102 -> 0.0.0.0 [1/0/pim/querier/leaf] 135.1.2.102 -> 135.1.2.2 [1/0/pim/querier] 135.1.2.102 -> 135.1.2.3 [1/0/pim/querier] 135.1.3.102 -> 135.1.22.98 [1/0/tunnel/querier]

pim-dvmrp-gw>mrinfo 135.1.22.98135.1.22.98 [version mrouted 3.8] [flags: GPM]: 172.21.32.98 -> 172.21.32.191 [1/1] 172.21.32.98 -> 172.21.32.1 [1/1] 135.1.22.98 -> 135.1.22.102 [1/1/querier] 135.1.22.98 -> 135.1.3.102 [1/1/tunnel]

Using the “mrinfo” Command

Both Ends SeeEach Other

Page 475: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

475

PIM-DVMRP Troubleshooting

Verifying the DVMRP tunnel Verifying DVMRP route exchangeVerifying DVMRP route exchange

Page 476: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

476

Verifying DVMRP Route Exchange

Using “show ip dvmrp route”pim-dvmrp-gw# show ip dvmrp routeDVMRP Routing Table - 8 entries

130.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

135.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

135.1.22.0/24 [0/2] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

171.69.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

172.21.27.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

172.21.32.0/24 [0/2] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

172.21.33.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

172.21.120.0/24 [0/3] uptime 00:19:04, expires 00:02:12 via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]

Page 477: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

477

Verifying DVMRP Route Exchange

Using “debug ip dvmrp”pim-dvmrp-gw# debug ip dvmrp DVMRP debugging is onpim-dvmrp-gw#Mar 20 11:39:36.335: DVMRP: Aging routes, 0 entries expiredMar 20 11:39:41.271: DVMRP: Received Probe on Tunnel0 from 135.1.22.98Mar 20 11:39:45.335: DVMRP: Building Report for Tunnel0 224.0.0.4Mar 20 11:39:45.335: DVMRP: Send Report on Tunnel0 to 135.1.22.98Mar 20 11:39:45.335: DVMRP: 2 unicast, 8 DVMRP routes advertisedMar 20 11:39:47.335: DVMRP: Aging routes, 0 entries expiredMar 20 11:39:51.371: DVMRP: Received Probe on Tunnel0 from 135.1.22.98Mar 20 11:39:52.379: DVMRP: Received Report on Tunnel0 from 135.1.22.98

Page 478: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

478

Verifying DVMRP Route Exchange

pim-dvmrp-gw# debug ip dvmrp detailDVMRP debugging is onMar 20 11:42:45.337: DVMRP: Building Report for Tunnel0 224.0.0.4Mar 20 11:42:45.337: DVMRP: Report 130.1.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.22.0/24, metric 34, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 171.69.0.0/16, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.27.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.32.0/24, metric 34, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.33.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 172.21.120.0/24, metric 35, from DVMRP tableMar 20 11:42:45.337: DVMRP: Report 135.1.2.0/24, metric 1Mar 20 11:42:45.337: DVMRP: Report 135.1.3.0/24, metric 1Mar 20 11:42:45.337: DVMRP: Send Report on Tunnel0 to 135.1.22.98Mar 20 11:42:45.337: DVMRP: 2 unicast, 8 DVMRP routes advertised

Checking DVMRP Routes Being Advertised

Page 479: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

479

Verifying DVMRP Route Exchange

pim-dvmrp-gw# debug ip dvmrp detailDVMRP debugging is on... : DVMRP: Received Report on Tunnel0 from 135.1.22.98... : DVMRP: Origin 130.1.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 135.1.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 171.69.0.0/16, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 135.1.2.0/24, metric 34, metric-offset 1, infinity... : DVMRP: Origin 135.1.3.0/24, metric 34, metric-offset 1, infinity... : DVMRP: Origin 135.1.22.0/24, metric 1, metric-offset 1, distance 0... : DVMRP: Origin 172.21.27.0/24, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 172.21.32.0/24, metric 1, metric-offset 1, distance 0... : DVMRP: Origin 172.21.33.0/24, metric 2, metric-offset 1, distance 0... : DVMRP: Origin 172.21.120.0/24, metric 2, metric-offset 1, distance 0

Checking DVMRP Routes Being Received

Page 480: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

480

Advanced Troubleshooting

Troubleshooting Network State

Troubleshooting PIM-DVMRP Troubleshooting ATM P2MP Troubleshooting ATM P2MP

VCsVCs

Page 481: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

481

Multicast over ATM P2MP VCs

ATMATMNBMANBMACloudCloud

One p2mp VC/group performs multicast replication instead of the router

B’cast p2mp VC used when # Groups > max p2mp VC count

Use PIM Sparse mode p2mp VCs map group m

embership Fast Switched!!

A

DD

BB CC

Note: Only p2mp m’cast VCs for Router A shown for clarity.

Page 482: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

482

M’cast P2MP VC Troubleshootingrtr-a> show ip pim vc IP Multicast ATM VC Status ATM0/0 VC count is 5, max is 5 Group VCD Interface Leaf Count Rate 224.0.1.40 21 ATM0/0 2 0 pps 224.2.2.2 26 ATM0/0 1 0 pps 224.1.1.1 28 ATM0/0 1 0 pps 224.4.4.4 32 ATM0/0 2 0 pps 224.5.5.5 35 ATM0/0 1 0 pps

Page 483: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

483

rtr-a> show atm vc AAL / Peak Avg. Burst Interface VCD VPI VCI Type Encapsulation Kbps Kbps Cells StatusATM0/0 1 0 5 PVC AAL5-SAAL 155000 155000 96 ACT ATM0/0 2 0 16 PVC AAL5-ILMI 155000 155000 96 ACT ATM0/0 3 0 124 MSVC-3 AAL5-SNAP 155000 155000 96 ACT ATM0/0 4 0 125 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 5 0 126 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 6 0 127 MSVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 9 0 130 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 10 0 131 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 11 0 132 MSVC-3 AAL5-SNAP 155000 155000 96 ACT ATM0/0 12 0 133 MSVC-1 AAL5-SNAP 155000 155000 96 ACT ATM0/0 13 0 134 SVC AAL5-SNAP 155000 155000 96 ACT ATM0/0 14 0 135 MSVC-2 AAL5-SNAP 155000 155000 96 ACT ATM0/0 15 0 136 MSVC-2 AAL5-SNAP 155000 155000 96 ACT

Root P2MP VC with 3 Leaf RoutersRoot P2MP VC with 3 Leaf Routers

P2MP VC for which we are a LeafP2MP VC for which we are a Leaf

M’cast P2MP VC Troubleshooting

Page 484: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

484

show ip mroute 224.1.1.1 IP Multicast Routing Table Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT Timers: Uptime/Expires Interface state: Interface, Next-Hop or VCD, State/Mode (*, 224.1.1.1), 00:03:57/00:02:54, RP 130.4.101.1, flags: SJ Incoming interface: Null, RPF nbr 0.0.0.0 Outgoing interface list: ATM0/0, VCD 3, Forward/Sparse, 00:03:57/00:02:53

ATM P2MP VC information for GroupATM P2MP VC information for Group

M’cast P2MP VC Troubleshooting

Page 485: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

485

rtr-a> show atm vc 3

ATM0/0: VCD: 3, VPI: 0, VCI: 124, etype:0x0, AAL5 - LLC/SNAP, Flags: 0x650PeakRate: 155000, Average Rate: 155000, Burst Cells: 96, VCmode: 0xE000OAM DISABLED, InARP DISABLEDInPkts: 0, OutPkts: 12, InBytes: 0, OutBytes: 496InPRoc: 0, OutPRoc: 0, Broadcasts: 12InFast: 0, OutFast: 0, InAS: 0, OutAS: 0OAM F5 cells sent: 0, OAM cells received: 0Status: ACTIVE, TTL: 2, VC owner: IP Multicast (224.1.1.1)interface = ATM0/0, call locally initiated, call reference = 2vcnum = 11, vpi = 0, vci = 132, state = Active aal5snap vc, multipoint callRetry count: Current = 0, Max = 10timer currently inactive, timer value = 00:00:00Leaf Atm Nsap address: 47.0091810000000002BA08E101.444444444444.02Leaf Atm Nsap address: 47.0091810000000002BA08E101.333333333333.02Leaf Atm Nsap address: 47.0091810000000002BA08E101.222222222222.02

P2MP VC Opened by Group 224.1.1.1 P2MP VC Opened by Group 224.1.1.1

NSAP Addresses of Leaf Nodes NSAP Addresses of Leaf Nodes

M’cast P2MP VC Troubleshooting

Page 486: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

486

Troubleshooting Tools Basic Troubleshooting Advanced

Troubleshooting Case studiesCase studies

Agenda

Page 487: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

487

Troubleshooting Cheat Sheet Check IGMP membership on PIM DR on Last

Hop LAN Check RP address in (*,G) entry on the DR Check RPF interface to RP in (*,G) entry Repeat above check for (*,G) state on routers

along the shared tree, up to RP. Repeat above check for (S,G) state on routers

along the source tree, up to Source DR.

Page 488: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

488

PIM Timers The secret to understanding PIM is to watch

the timers. 3 minutes (3:30) is the “magic” number. Interface expiration timers are updated every

minute so if the expire timer goes below 2:00 the route is not being used.

Entry expiration timers are updated when data is forwarded so if the timer drops below 2:59, the source has stopped sending.

Page 489: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

489

Source Tree

In IOS a (*,G) entry is always created whenever a (S,G) entry is created.

The Source-tree may overlap the Shared-tree in which case the (*,G) entry will be non-NULL.

The Source-tree may be independent of the Shared-tree in which case the (*,G) entry will be NULL.

Page 490: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

490

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32

•(S,G) forwarding entry

NOTE: These uptimes indicate the receiver has always been present

PIM SM Source Tree

Page 491: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

491

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32

•(S,G) forwarding entry

Receivers have stopped joining

PIM SM Source Tree

Page 492: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

492

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: Serial1, Forward/Sparse, 00:04:28/00:01:32

•(S,G) forwarding entry

Data is not flowing

PIM SM Source Tree

Page 493: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

493

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

All Sources for this group will be forwarded out the olist

PIM SM Shared Tree

Page 494: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

494

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

This always points to the RP

PIM SM Shared Tree

Page 495: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

495

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

This is the next-hop to the RP from “sh ip RPF”

PIM SM Shared Tree

Page 496: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

496

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

The entry has been up for this long. Note the uptime of the olist

PIM SM Shared tree

Page 497: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

497

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

All receivers for the entry may have left

PIM SM Shared tree

Page 498: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

498

(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S Incoming interface: Serial1, RPF nbr 171.68.28.140, Outgoing interface list: Serial0, Forward/Sparse, 00:04:28/00:01:32

•(*,G) forwarding entry

A sparse-mode group must have an RP

PIM SM Shared tree

Page 499: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

499

(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: RP Incoming interface: Serial0, RPF nbr 171.68.28.190 Outgoing interface list: NULL

•(S,G,R) forwarding entry

Points toward the RP!!!!

PIM Shared Tree

Last-hop router is sending (s,g,r) prunes

Note R-flag

Page 500: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

mtrace

• Shows:– Multicast path from source to receiver.

» Similar to unicast “trace” command» Trace path between any two points in network» TTL Thresholds & Delay shown at each node

• Troubleshooting Usage:– Find where multicast traffic flow stops.

» Focus on router where flow stops

– Verify path multicast traffic is following.» Identify sub-optimal paths.

Page 501: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

501

Multicast Dist. Tree

Mtrace Packet

src dest

mtrace re

quest

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

Addsmtrace

data

mtrace response

Unix Workstationor

Cisco Router

Last-hopRouter

First-hopRouter

Mtrace Packet Flow

mtrace/mstat—How it works

Note: Mtrace packets use specialIGMP packets with IGMP Type codes of 0x1E and 0x1F.

Page 502: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

mtrace

dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>

dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43Type escape sequence to abort.Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43From source (?) to destination (bwilliam-ss5.cisco.com)Querying full reverse path... 0 bwilliam-ss5 (171.68.37.121)-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms-5 bloom-iptv-svr (172.17.67.43)dallas-gw>

Page 503: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

mstat

• Shows:– Multicast path in pseudo graphic format.

» Trace path between any two points in network» Drops/Duplicates shown at each node» TTLs & Delay shown at each node

• Troubleshooting Usage:– Locate congestion point in the flow.

» Focus on router with high drop/duplicate count» Duplicates indicated as “negative” drops

Page 504: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

mstatdallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source

dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 547 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 547 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop -409 ms -11/168 = --% 16 pps 0/67 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop 379 ms -9/170 = --% 17 pps -3/67 = --% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 28 ms -3/195 = --% 19 pps 0/70 = 0% 7 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 196 19 pps 70 7 pps171.68.37.121 171.68.86.194 Receiver Query Source

Page 505: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

mstatdallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source

dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43 Source Response Dest Packet Statistics For Only For Traffic172.17.67.43 171.68.86.194 All Multicast Traffic From 172.17.67.43 | __/ rtt 399 ms Lost/Sent = Pct Rate To 224.2.156.43 v / hop 399 ms --------------------- --------------------172.17.67.33 171.68.39.28 bloom-mnlab | ^ ttl 0 v | hop 119 ms 77/694 = 11% 69 pps 0/65 = 0% 6 pps171.68.39.1 171.68.27.2 bloomington-mn-gw | ^ ttl 1 v | hop -150 ms 395/609 = 65% 60 pps 44/65 = 68% 6 pps171.68.27.1 171.68.86.193 wan-gw4 | ^ ttl 2 v | hop 30 ms -8/39 = --% 3 pps -1/21 = --% 2 pps171.68.86.194 171.68.37.1 dallas-gw | \__ ttl 3 v \ hop 0 ms 39 3 pps 22 2 pps171.68.37.121 171.68.86.194 Receiver Query Source

Page 506: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

506

Debugging Auto-RP Operation

Understand the Auto-RP mechanisms This is the fundamental debugging tool for problems with Auto-This is the fundamental debugging tool for problems with Auto-

RP!!!RP!!! Verify Group-to-RP Mapping Caches

First on the Mapping Agents Other routers will learn Group-to-RP mapping info from these routers

If not correct, use debug commands to see what’s wrong Make sure all MA’s have consistent Group-to-RP information

If not, watch for TTL Scoping problems Then on other routers

If info doesn’t match MA, there is a problem distributing the information

Use show and debug commands to find where the break is

Page 507: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

507

Debugging Auto-RP Operation Insure Auto-RP group state is correct

Should normally be in Dense mode Watch out for mixed DM and SM conditions

Can occur when Static RP’s are also defined Always ‘deny’ Auto-RP groups on Static RP

configurations Use ‘Accept-RP’ filters on all routers as insurance

Watch out for DM problems in NBMA networks

(See Module 7 for details)

Page 508: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

508

Debugging BSR Operation Understand the BSR mechanisms

This is the fundamental debugging tool for problems with This is the fundamental debugging tool for problems with BSR!!!BSR!!!

Verify Group-to-RP Mapping Caches First on the BSR

Other routers will learn Group-to-RP mapping info from this router

If not correct, use debug commands to see what’s wrong

Then on other routers If info doesn’t match BSR, there is a problem distributing

the information Use show and debug commands to find where the break

is

Page 509: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

Case Studies

Page 510: Trouble Shooting CERNET NOC Pengfei Li. 2 Contents Trouble Shooting Tools Trouble Shooting BGP Trouble Shooting OSPF Trouble Shooting IS-IS Trouble Shooting.

510

THANKS