Jim Mallory, Supervisor of Network OperationsSaginaw Intermediate School District

Troubleshooting Switches, Firewalls, and Wireless

•You will be able to use some of the tools built-in to your network infrastructure gear to troubleshoot common problems

•How you benefit: Increase your ability to solve day-to-day issues on your network or, if the need arises to engage vendor support, you will have a base level of knowledge to help them with the issue

• I have been troubleshooting Enterprise level networks for 25 years. (Ethernet, Token-Ring, IP, SNA, AppleTalk, IPX, X.25, Frame-Relay, and ATM, HDLC/SDLC). The majority of the time troubleshooting very low level protocol and hardware issues with specialized hardware.

Introduction (what I hope to accomplish)

• Network Switches▫ Identifying what port a device is on if you know the IP address

Finding the MAC address Identifying the device

▫ Finding a wireless MAC address ▫ Troubleshooting fiber connections▫ Troubleshooting cable issues

• Firewall Troubleshooting▫ Setting up a packet capture on a Cisco ASA with ASDM and Wireshark▫ Graphing CPU, Memory, and Sessions

• Wireless Troubleshooting▫ How to check signal strength▫ Spectrum Analysis▫ Packet Capture with Wireshark

Presentation Outline

• We know the IP address▫ We need the MAC address. This is what the switch tracks▫ We can get the MAC address two ways

The switches ARP table, this sometimes (rarely) works HP ProVision: show ip arp <ip address>

DHCP Server Records: This always works▫ We now know the MAC address

If I know the building, I will start at the building level core (MDF) switch If I don’t, I start at the district core HP ProVision: show mac-address aabbcc-ddeeff Follow-up with a show lldp info remote-device or show cdp neighbor to determine if the device on the

far end is a switch, AP, or the device itself. Some devices don’t support lldp/cdp so YMMV. If a switch, telnet (ssh) into that switch; if an AP log into the controller / AP Repeat these steps until you get to an AP or the device itself

Network Switches – How to find what port a device is on

Finding the Wireless MAC Address

•Current fiber optic drivers have built-in DOM (Diagnostics On Module) capability that can be used to do some basic troubleshooting

•HP ProVision Command: show interface transceiver <port> or <slot/port> detail

• Interested in Rx Power▫0mw, 0dBm is bad – not receiving light from far side▫Intermittent Connections: Could be that your Rx power is marginal but you

will need to know what the minimum amount of power the module needs.▫Should be able to find it on your fiber optic vendors support site. ▫Example: 10Gbase-LR requires -14 dBm

Troubleshooting Fiber Links

•Some new “Enterprise” class switches have built in Time Domain Reflectometers (TDRs) that can be used for cable diagnostics

•This is usually disruptive as it breaks Ethernet connectivity while the switch is performing the test

•Ubquity EdgeSwich OS: cablestatus slot/port

Cable Testing

•Packet Capture with Cisco ASDM and Wireshark•CPU, Memory, Sessions graphing with ASDM•Firewall problems sometimes aren’t strictly about “bandwidth”, could

be other issues such as the number of active sessions or the number of new sessions being setup per second.

Firewall Troubleshooting

• Open Source packet capture and decode tool (started life as Ethereal)

• You really need to understand the low level protocols to get the most out of this tool

• Runs on Windows, Mac OS X, and other *nixes (Linux)

• Similar functionality to commercial packet analysis tools at a much, much, lower cost.

• Also can do wireless sniffing with the right adapter (AirPCAP, next session)

• Available at www.wireshark.org (along with some training materials)

A Little Bit on “Wireshark”

•Signal Strength▫How to determine via the controller

•Spectrum Analysis▫Via the Controller or AP (this is usually disruptive)▫Dedicated Spectrum Analysis Tools

Metageek

Wireless Troubleshooting

• Two ways to measure• RSSI (Relative Signal Strength Indication)

▫ At least -67 dBm▫ One Ruckus Engineer stated that with high

density 1:1 environments, high 50s may now be the new “ideal”

• SNR (Signal to Noise Ratio)▫ Calculated from the difference between the

RSSI of the wireless device as measured by the AP/Controller and the noise floor again as measured by the AP/Controller

▫ HP considers the minimum SNR for what it considers a “low” quality signal is 16 dB

Signal Strength

• Some Wireless systems allow you to put an AP into spectrum analysis mode for troubleshooting

• This is almost always disruptive as the AP in this mode will no longer service wireless clients

• Dedicated software / hardware based analysis tools are available▫ If you manage any kind of substantial wireless

install (greater than a dozen APs) or multiple installs (ISD). You need this tool or at least access to someone who does

▫ NOT A SITE SURVEY TOOL▫ Does require special USB spectrum analysis

cards along with your built-in WiFi card to capture BSSID information

Spectrum Analysis

• Hardware▫ Microsoft Surface Pro 2 (Windows 8.1 Update 1)▫ i5 processor▫ 4GB of RAM▫ 128GB of SSD▫ Built-in wireless card (used to collect BSSID info)▫ Two USB spectrum analyzers

WiSpy DBx 2.4Ghz / 5Ghz combo spectrum analyzer WiSpy DB2.4x 2.4Ghz dedicated spectrum analyzer

▫ AirPCAP – Packet capture USB radio• Software

▫ Metageek Channelyzer Pro (Spectrum Analyzer software)▫ Metageek EyePA (Basic Packet Capture software)▫ Wireshark (Advanced Packet Capture software)

My Wireless Toolkit

Channelyzer Demo

•You may be able to setup your AP to capture packets and send them to Wireshark for further decoding

•Will not capture 802.11 radio information (just higher level protocols)•To capture packets at the 802.11 radio level. You will need the AirPCAP

software and compatible card.• I use the Riverbed card.

Wireless Packet Capture and Analysis with Wireshark

•Any questions about any of this?•You can always email me at jmallory@sisd.cc

Wrap-up

www.wireshark.org – Wireshark download, Wiki, Videos, etc.www.wlanpros.com – Excellent resource for everything 802.11www.revolutionwifi.net – Blog detailing an 802.11 design methodology. (wouldn’t have to do so much troubleshooting if the things were designed properly)

http://forums.juniper.net/jnet/attachments/jnet/Day1Books/5/8/Junos%20Monitoring%20and%20Troubleshooting.pdf – Juniper “Day One” book on monitoring and troubleshooting. Requires J-Net Membership (Free) – Illustrates with JunOS commands, but basic concepts are the same.

https://h30590.www3.hp.com/product/HP+Networking+and+Cisco+CLI+Reference+Guide+-+Version+2-Paperback-8409

Excellent resource to convert the ProVision commands I used to “Ciscospeak”. Also a great resource if you’re an old Cisco guy like me and need to learn HP. The PDF is FREE!

Recommended Websites and Downloads