“IP Network Troubleshooting“ Part 2

Wayne M. Pecena, CPBE, CBNE Texas A&M University

Educational Broadcast Services - KAMU

November 2015

" IP Network Troubleshooting – Part 2"

Advertised Presentation Scope:

Successful troubleshooting of any system relies upon applying a logical

approach with knowledge of the technology involved, and maybe a little luck

when experience is lacking. This webinar will focus upon applying a logical

approach to troubleshooting an IP network in a LAN environment with a basic

understanding of IP networking principals by following the data flow layers of

the OSI model. A focus will be placed upon utilization of techniques and use

of common open-source tools to identify and isolate network connectivity and

performance abnormalities.

Webinar Goals:

Continue Structured Troubleshooting Approach (from part 1)

Understand the Basics the OSI Model Layers 3-4 in an IP Network

Understand & Apply Techniques to Verify Layer 3-4 Functionality

Understand How to Get Started with Wireshark Protocol Analysis

2

Today’s Outline:

• Refresh Takeaways from Part 1

• Network & Session Layer Functions & Verification

• Understanding ping & traceroute

• Getting Started with Wireshark

• Taping Into the Network

• Wireshark Capture & Filtering

• Takeaways, Questions, and Maybe Some Answers

3

Takeaways Points from Part 1

4

5 Things Required To Build a Network

• Send Host

• Receive Host

• Message or Data to Send Between Hosts

• Media to Interconnect Hosts

• Protocol to Define How Data is Transferred

5

Protocols

Send Host Receive Host

MediaMedia

DATA

The Structured Troubleshooting Process

6

Problem Identification

Problem Diagnosis

Problem Resolution

Develop a

Structured Approach

To

Resolution

Avoid

an

Unstructured Approach

ProblemIdentification

ProblemRe-Creation

Localize & Isolate

Problem

FormulateResolution

Plan

DocumentResolution

Provide FeedbackTo Users

ImplementResolution

Plan

VerifyResolutionResolved

The OSI Model Open Systems Interconnection (OSI) Model

7

Networking

Focus

Open Systems Interconnection “OSI” Model

8

Part

2

Focus

Transport

Physical

Data Link

Network

4

1

2

3

Manages End-End Connections:

TCP, UDP, & Flow Control

Interfaces to Physical Network, Moves Bits Onto &

Off Network Medium

Provides Network Access Control, Physical

Address (MAC), & Error Detection

Provides Internetwork Routing (path)

Provides Virtual Addressing (IP)

Network Abnormalities

• Categories of Abnormalities: – No Connectivity

• Cable Fault

• Blocked or Failed Switch Port

• Failed Host NIC

– Intermittent Connectivity

• Cable Fault

• Failed Switch Port

• Failed Host NIC

• Duplex Mismatch

– Poor Performance

• Cable Fault

• Failed Switch Port

• Failed Host NIC

• Duplex Mismatch

9

Common Layer 1 Faults • Copper Cabling Aspects:

– Continuity Problems (open, shorts, crossed)

– Wrong Cable for Application

– Improper RJ-45 Installation

– Excessive “Un-Twist”

– Excessive Connections (connector blocks)

– Excessive Segment Length

• Fiber Cabling Aspects: – Damaged Fiber

– Improper Connector Installation

– Dirty Connectors

– Component Aging

10

11

12

Optical Power Guidelines:

Stay Within +/- 4 dB

Receive Power Generally Within:

-27 to -8 dB

Design “Sweet Spot”: (66%)

-17 to -23

Launch Power (Transmit) Selected

For the Required Optical Budget

Common Layer 2 Faults

• Failed or Intermittent Host NIC

• Failed or Intermittent Switch Port

• Duplex Mismatch

• Excessive Errors

13

Use

“Managed”

Switch

Port

Capabilities

To

Verify

Operation

Cisco Switchport Metrics

14

Network Documentation “Network Cartography”

15

Documentation Excuses

• “I don’t need to put anything in writing, I’ve done this kind of system a million times.”

• “I don’t need a roadmap. I can just tell where things need to go.”

• “My company doesn’t have the money to invest in creating those documents.”

16

Source: Summarized from InfoComm Blog 11/6/15

Takeaway Points & Concepts – Part 1

• Establish a “Structured” Troubleshooting Approach

• Use the OSI Model as a Guide – Verify Layer 1 Physical Connectivity

– Verify Layer 2 Connectivity is Error Free

• 80% of True Network Problems is Physical Infrastructure Based – Standards Not Properly Applied

– Guidelines Not Adhered To

• Don’t Loose Sight of 100m Ethernet Segment Limit!

• Network Documentation & Baseline Performance Metrics Are Essential to Efficient Network Problem Resolution!

17

Network & Session Layer Functions & Verification

18

IPv4 Packet – Layer 3 RFC 791

19

Version

(4)

Header

(4)

Precedence / Type

(8)

Length

(16)

Identification

(16)

Flag

(3)

Offset

(13)

Time to Live

(8)

Protocol

(8)

Header Checksum

(16)

Source IP Address

(32)

Options & Padding

(0 or 32)

Destination IP Address

(32)

Packet Payload

(Transport Layer Data)

32 bits

20

Bytes

2-Part IPv4 Address

20

192

32 bit IP Address

1100000010101000110010011111110

168 100 254

11000000 10101000 1100100 11111110

Subnet

Mask

Determines

Network

Address

Host

Address

Octet 1 Octet 2 Octet 3 Octet 4

4 Bytes

VLSM • Allows Mask to Be Determined on a “Bit Basis”

– Classful Addressing Specifies Network / Host Boundary

– Classless Addressing Allows Network / Host Boundary to Be Specified at an Individual Bit

21

Octet 1 Octet 2 Octet 3 Octet 4

Octet 1 Octet 2 Octet 3 Octet 4

A B C

19 Subnet Mask Bits = 255.255.224.0

Network Host

Network Host

IPv4 Address Subnet Mask Example “VLSM” - Each IP Address Must Have a Subnet Mask to Define the Network and the Host

32 Bit Subnet Mask

Expressed in Decimal as (4) 8-bit Octets using “Doted Decimal Notation”

IP Address: 192.168.100.254 /19

192.168.100.254 /19 or 255.255.224.0

11000000.10101000.00000001.01100100

11111111.11111111.11100000.00000000Network Host

Is My IP Address Correct? Reverse Engineering an IP Network

23

You Need to Know: Useable IP Address Range?

See .pdf Handout for Further Study

TCP Basics Transmission Control Protocol

• “Connection – Oriented” Protocol – Connection Establishment

– Segmentation & Sequencing

– Acknowledgement

– Flow Control or Windowing

• Guaranteed Or Reliable Data Delivery – Acknowledgment of Packet Receipt

– Retransmission Occurs if Packet Not Received

• High Overhead

• Requires Establishment of a “Session”

• TCP Windowing Feature – Dynamic Window Sizing

– “Slow-Start”

24

TCP 3-Way Handshake

25

Host 1 Host 2

SYN

SYN + ACK

ACK

Host 1 Sends

Synchronize Message

to Host 2

Host 2 Responds With

Acknowledgement

Plus Sends It’s Own

Synchronization

Message to Host 1Host 1 Completes the

3-Way Handshake By

Sending

Acknowledgement to

Host 2

Host 1 Initiates

Connection to Host 2

The TCP Session Summary

26

SYN + ACK

Time

Network

SYN

ACK

FIN

FIN

ACK

ACK

ACK

Connection

Closed

Listen

SYN Sent

SYN Received

Connection

Established Connection

Established

Connection

Closed

FIN Wait 1

FIN Wait 2

CLOSE Wait

Last ACK

ACK

ACK

Data Segment 1

Data Segment 2

Data Segment 3

TCP Sequencing

27

Host 1

Host 2

Sequence Number 1

Sequence Number 1501

Receive ACK

Sequence Number 3001

Sequence Number 4501

Receive ACK

Receive 1 – 1500

Receive 1501 – 3000

Send ACK 3001

Receive 3001 – 4500

Receive 4501 – 6000

Send ACK 6001

1500 bytes

1500 bytes

1500 bytes

1500 bytes

TCP Connection

Established

Window Size = 3000

UDP Basics User Datagram Protocol

• “Connectionless” Protocol

• Simple or Lightweight, but Inherently Unreliable

• “Best Effort” Data Delivery

• Low Overhead, Thus Low Latency

• Why Use?

– Required for Real-Time Applications: • VOIP or “Video Over IP” or “Voice Over IP”

• AOIP or Audio Over IP”

– Latency More Detrimental Than Data Loss

28

UDP Session

29

Network

SYN

SYN + ACK

ACK

Data

Data

Data

Time

Data

Data

TCP Used to

Establish UDP

Session

TCP and UDP Headers

30

Connection Termination

31

Host 1 Host 2

FIN

FIN + ACK

ACK

Host 1 Sends Finish

Message to Host 2

Host 2 Responds With

Finish Plus Sends It’s

Own Synchronization

Message to Host 1

Host 1 Completes the

Termination By

Sending

Acknowledgement to

Host 2

Host 1 is Ready to

Terminate Connection

Understanding ping & traceroute

32

ICMP Internet Control Message Protocol

• Network Layer Based – RFC 1256 – The “Tattle Tale” Protocol

• Unique IP Based “Message” (IP protocol #1)

• Message Originated by a Layer 3 Device “Router”

• Message Sent to a Host or Another Router

• Common Messages: – Destination Unreachable

– Buffer Full

– Hops or Time Exceeded (TTL)

• Common Use by Network Utilities: – Ping

– Traceroute

ICMP in Detail

34

“ping” Packet Internet Groper

35

Send Hosts Sends ICMP “echo request”

Destination Host Replies ICMP “echo reply”

Round-Trip Times Returned

Be Aware of Command Line Options

“traceroute” RFC 1812

• The Most Widely Used Network Diagnostic Tool

• The Most Widely Misunderstood Network Diagnostic Tool

• How? – Send Host Transmits 3 UDP Packets to Receive Host With TTL = 1 (port typical 33434)

– First Hop Router Sends icmp TTL Exceeded

– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 2

– Second Hop Router Sends icmp TTL Exceeded

– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 3

– Third Hop Router Sends icmp TTL Exceeded

– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 4

36

Send Host

Receive Host

icmp TTL exceeded

icmp TTL exceeded

icmp TTL exceeded

icmp destination port unreachable

TTL=1 TTL=4 TTL=3 TTL=2

Understanding “traceroute”

37

Windows

Command

Screen

“tracert”

PingPlotter

What Is

Up?

Limitations of “ping” & “traceroute” • ICMP May Be Blocked Within Networks

• Routers May Limit ICMP Processing (interfaces limited)

• Realize Layer 2 Devices Will Not Be Seen

• Protocol Utilized by traceroute Can Impact Results (UDP, ICMP, TCP)

• Understand: – traceroute Forward Path Route is Displayed (return path may be different)

– traceroute returns Round-Trip Latency

• Understand Traceroute Latency: – Latency Increase May Not Be Significant

– Latency Increase Must Continue Increasing for Additional Hops To Be of Concern

38

Cisco Routing Verification

39

Useful “Cisco” IOS Troubleshooting Commands:

R1# show ip protocols

R1# show ip route

R1# show ip route 66.39.27.70

R1# ping 66.39.27.70

Getting Started with Wireshark

40

What is “Wireshark?” • “Open Source” Protocol Analyzer Application

• Often Referred to as a “Sniffer”©

• Developed in 1998 as “Ethereal”

• Renamed in 2006 Due to Trademark Issues

• Analyses of “Live” & “Recorded” Network Activity

• Useful To: – Isolate performance issues

– Understand application interaction

– Benchmarking

Gerald Combs

Wireshark Developer

Obtain & Install “Wireshark”

• Available for Windows, Mac OSx, & Linux

• Download at: www.wireshark.org

• Include Libraries:

– WinPcap

– libpcap

NIC “Promiscuous” Mode

Network

Interface

Controller

“Listens” for

It’s MAC Address

& Broadcasts

“Promiscuous” Mode

Processes All Frames Received

Transport

Network

Data Link

Physical

Upper Layers

Network

Packet Capture“pcap”

Packet Analyzer Ap“Wireshark”

All TX & RXFrames

pcap ”packet capture”:

API (application programming

interface) for capturing network

traffic:

libpcap for Unix/Linux

WinPcap for MS Windows

Taping Into the Network

44

Where to Tap?

• Problem Nature Often Determines:

– At Problem Host

– At Destination Host

– Mid-Network Locations

• Accessibility May Also Drive Tap Point

Taping Into Ethernet

• Can Be Challenging!

• Where to Tap?

• How to Tap?

– Physical Passive Tap

– Active Tap

– Ethernet Switch Port Mirror

47

Application

Session

Presentation

Transport

Physical

Data Link

Network

Application

Session

Presentation

Transport

Physical

Data Link

Network

Physical

Data Link

Physical

Data Link

Physical

Data Link

Physical

Data Link

Physical

Data Link

Physical

Data Link

Network Network

Layer 2

Device Layer 2

Device Layer 3

Device

Tap Devices

UTP Taps

Optical Taps

HostA

HostB

HostD

HostC

Wireshark Host

Ethernet Switch Port Mirror or “SPAN Port” Switched Port Analyzer Port

Goal – Observe Traffic Between

Host A & Host B on Wireshark Host

“Span” Port

Ops! – Ethernet Switch

Isolates Network

Traffic

Define Source Port(s)

Designate a “Span Port”

config t

monitor session 1 source interface fa0/1

monitor session 1 source interface fa0/23

monitor session 1 destination interface fa0/14

exit

Wireshark Capture & Filtering

50

Capturing Network Traffic

51

Captured Packet(s) List

Selected Header

Data Decoded

Payload Data Decoded Hexadecimal & ASCII

Wireshark Views

packet 192 selected

Header Details Displayed

Payload Data Decoded

Filtering

• Capture Filters – Selectively Capture Packets

– Pre-Capture Configuration

– Minimizes Captured Data

• Analysis Filters – Applied When Viewing

– Allows Focusing on an Attribute(s)

– All Data is Retained

Filter Example

Wireshark Example “Benchmark Network Activity”

56

Wireshark Example “ping www.sbe.org”

57

Wireshark Example

58

“audio stream example – udp filter & modify column headings”

Wireshark Example “TCP/IP Window”

59

100 101 102 103 104 105 106

Bytes Receive

Is Ready to Accept

107 108 109 110 111 112

TCP Receive Window

BYTES Sent

NOT AcknowledgedBYTES Sent &

AcknowledgedBytes Receiver

Is NOT Ready to Accept

RFC 1072 & RFC 1323

Takeaways, Questions, and Maybe Some Answers

60

Takeaway Points & Concepts – Part 2

• Understand Limitations of “ping” & “traceroute”

• Protocol Analysis Is Essential to “See” Network Activity

• “Wireshark” Is The Most Popular Protocol Analyzer

• Understanding the OSI Model & TCP/IP Protocol Action is Essential to Understanding Wireshark Results

• Pre or Post Capture “Filtering” is Essential to Find the Needle in the Haystack

– Capture Filters

– Display Filters

• Wireless Is Also Supported by Wireshark – Think “Layers 1 & 2”

• “Flat Broke” is Often Easier to Fix Than Performance Issues!

• The Network is Commonly Blamed for Performance Issues: In Reality < 5% – You Are Often Proving the “Network is Innocent” (Gartner Research)

61

There is Always More to Know!

62

Don’t Miss IP Network Troubleshooting Part 3!

February 23, 2016 – 2pm ET

Focused on Protocol Analysis in the Broadcast

Plant

References – Further Study

https://wiki.wireshark.org/

Graphical “Traceroute” Utility

http://www.pingplotter.com Free & Paid Versions

64

65

Thank You for Attending! Wayne M. Pecena Texas A&M University wpecena@sbe.org 979.845.5662

66

? Questions ?

4