Vi Networking Adv Troubleshooting

8/2/2019 Vi Networking Adv Troubleshooting

1/79

vmware.com/go/networking

VI3 Networking:Advanced Troubleshooting


2/79

22 vmware.com/go/networkingvmware.com/go/networking

ESX Networking Architecture

Physical NICs

Virtual Switch

Virtual NICs

VMs Virtual NIC

Flexible

Enhanced

Virtual E1000

Vswif for the ServiceConsole

VMkernel uses vmknic

VMkernel TCP/IP Stack

PhysicalSwitches

Hardware

ESXServer

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS

VMKernel TCP/IP Stack


3/79


Agenda

Basic TroubleshootingHow to isolate problems?

What tools are available for troubleshooting?

Troubleshooting Scenarios

Step-by-step guide on how to troubleshoot some specificnetworking problems


4/79

vmware.com/go/networking

Basic Troubleshooting

Techniques


5/79


Isolate the problem

Troubleshoot one component at atime

Physical NICs

Virtual Switch

Virtual NICs

Physical Network

Tools for troubleshooting

VI

Command Line Utilities

Third party tools

Ping and traceroute

Traffic sniffers and Protocol Analyzers

Wireshark

Logs

Hardware

VMKernel

VSwitch

ESX Server


6/79


Isolating Network Problems: Physical NICs

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS


What to look for?

Where to look?


7/79


Physical NICs: What to look for?

Does the device show up?

Is the driver loaded?

Physical properties of the link

Link State

Link Speed

Duplex Setting

MTU settings

Is the NIC connected to where you want itto be connected?

Is the NIC working?Is the NIC transmitting and receiving packets?

Is the NIC dropping any packets?

esxcfg-nics

ifconfig insideService Console

VI Client

Network Hints

Cisco DiscoveryProtocol (CDP)

ifconfig insideService Console

esxtop/resxtop

esxcfg-info


8/79


Getting information about the physical NICs

VI Client provides basic information about the physicalNICs

Type of NIC Link Status Connections Network Hint


9/79



esxcfg-nics allows you to set or get physical NICsettings via the command line


10/79



The - l option lists the nics in the system and theirsettings

Link State

Speed

Duplex

MTU


11/79


Where is the physical NIC connected to?

Just follow the cable

OR

Use CDP and Network Hints


12/79


Cisco Discovery Protocol

Periodic exchange ofinformation

Physical switch port a vmnic isconnected to

vSwitch a physical switch port is

connected toDuplex and speed settings


13/79


Cisco Discovery Protocol

CDP is enabled by default in listening mode

On ESX Server 3.5, it is possible to configure CDP also inadvertising mode

Enabled/disabled only via command line with

esxcfg-vswitch B

States Listen

Advertise

Both

Down

Verify the setting with

esxcfg-vswitch b


14/79



esxtop provides system-wide real-time trafficinformation

For ESXi use resxtop utility provided in the RCLI

Type n to switch to the network utilization screen

Output of esxtop


15/79



On ESX Server 3.5, running ifconfig inside Service Console providesinformation valuable for troubleshooting

Output of ifconfig


16/79



esxcfg-info provides detailed information about thesystem

Use esxcfg-infon for network information

Redirect the output of esxcfg-info to a file

Look for the Physical Nic section


17/79



Output of esxcfg-info


18/79



Search for vmnicX in theoutput of esxcfg-info



19/79


Summary: Getting information about the physical NICs

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS


VI Client

esxcfg-nics

esxtop/resxtop

ifconfig

esxcfg-info

CDP


20/79


Isolating Network Problems: Virtual Switch

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS



21/79


Virtual Switch: What to look for?

vSwitch and Portgroup ConfigurationUplinks

VLAN Setting

Layer 2 Security Policies

NIC Teaming Configuration

Is the traffic flowing through thevSwitch?

Is the vSwitch dropping any packets?

esxcfg-vswitch

esxcfg-info

VI Client

esxtop/resxtop

esxcfg-info


22/79


Getting information about the vSwitch: VI

VI: Virtual SwitchConfiguration

VI: PortgroupProperties


23/79


Getting information about the vSwitch: esxcfg-vswitch

esxcfg-vswitchAn interface for adding, removing, and modifying virtual switchesand their settings

Output of esxcfg-vswitch -l


24/79


Getting information about the vSwitch: esxtop

esxtop provides system-wide real-time trafficinformation

For ESXi use resxtop utility provided in the RCLI

Type n to switch to the network utilization screen

Real Time TrafficInformation

Output of esxtop


25/79


Getting information about the vSwitch: esxtop

System Running FloodPing to the vmknic

Hardware

ESX

Server

VMKernel

ServiceConsole

VMkernel

PhysicalSwitch

vmnic3

vmnic4

vmknic traffic is goingthrough vmnic4

Output of esxtop


26/79


Getting information about the vSwitch: esxcfg-info

esxcfg-info provides information in greater detailConfiguration information



27/79


Getting information about the vSwitch: esxcfg-info

esxcfg-info provides information in greater detailCumulative traffic information for each port on the vSwitch

Information about VMkernelPort

Information about Uplink Port(vmnic4)


28/79


Summary: Getting information about the vSwitch

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS


VI Client

esxtop/resxtop

esxcfg-vswitch

esxcfg-info


29/79


Isolating Network Problems: Virtual NICs

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS



30/79


Virtual NICs: What to look for?

Does the device show up?

Is the driver loaded?

Physical properties of the link

Link State

MTU settings

Is the vNIC connected to the correctportgroup?

Portgroup using the correct uplink

Portgroup with the correct security properties

Is the NIC working?

Does the NIC have an IP address?Is the NIC transmitting and receiving packets?

Is the NIC dropping any packets?

esxcfg-vswif

esxcfg-vmknicGuest specific utilities

Linux

ifconfig

lspci

Windows

Device Manager

VI Client

.vmx file

esxcfg-info

Guest specific utilities

Linux

ifconfig

Windows

Network Connections

esxtop/resxtop

esxcfg-info


31/79


Getting information about the vNIC

VI: Virtual Machine

Properties

.vmx file

VMs Connection Information


32/79



esxcfg-vswif

An interface to configure Service Console NIC

esxcfg-vmknic

An interface to configure VMkernel NIC

Output of esxcfg-vswif -l

Output of esxcfg-vmknic -l


33/79



Output of esxtop


Search for the portID of the vNIC inthe esxcfg-info

output

Look for Rx/Txinformation for the

vNIC you areinterested in

Cumulative Traffic

Information

Real time trafficinformation


34/79


Summary: Getting information about the vNIC

PhysicalSwitches

Hardware

ESX

Server

VMKernel NICVSwitch

VMKernel

VMotion iSCSINFS


VI Client

Guest Utilities

esxtop/resxtop

esxcfg-info


35/79


Sniffing For Trouble

Sniff for packets at different

layers for isolationPhysical Switch Port Level

vSwitch Level

VM Level

Look for

Lost PacketsLarge number of packetretransmissions

Anomalies reported by protocolanalyzers like Wireshark etc.

Look for patterns

Are packets of a certain type causingproblems?

Are packets of a certain size causingproblems?

Hardware

ESX

Serv

er

VMKernel

PhysicalSwitch

VSwitch

Mirrored

Port

Capture packettraces inside

the VM

Capture packettraces on the

vSwitch


36/79


Collecting Network Traces on the vSwitch

Hardware

ESX

Server

VMKernel

VSwitch

Set the VLAN ID of theService Console portgroup

to 4095

Enable promiscuous modefor the Service Consoleportgroup

VM A on VLAN 106

Run tcpdumpi vswifXin the Service Console Running

tcpdumpivmnic0 wont

work!

VLAN 106 Packet For VM AVLAN 106 Packet For VM A VLAN 106 Packet For VM A


37/79


Collecting Network Traces on the vSwitch

Hardware

ESX

Server

VMKernel

VSwitch

VM A on VLAN 106

Create a portgroup

Set the VLAN ID of the

portgroup to 4095

Enable promiscuous modefor the portgroup

Run Wireshark in theVM

VLAN 106 Packet For VM AVLAN 106 Packet For VM A VLAN 106 Packet For VM A


38/79


Logs on ESX

VMkernel logs

/var/log/vmkernelfor ESX

/var/log/messagesfor ESXi

VM logs

vmware.logfile in the VMdirectory

Service console logs

/var/log/messagesfor ESX

Also check the guest OSlogs for any errors

Hardware

ESX

Serv

er

VMKernel

VSwitch

VMkernel Logs

VMkernel Logs

VMkernel Logs

VM Logs

Guest OS Logs

Service Console Logs


39/79


40/79


Signs of trouble

Basic connectivity problems

No network connectivity on some or all of the VMs on a vSwitch

Flaky network connection

Connection timeouts

Intermittent loss of connectivity


41/79


Problem

Noneof the VMs on my ESX box have network

connectivity


42/79


43/79


Step 1: Check inside the VM

Is the network interface in the guest up and does it havean IP address?

Use OS specific utilities to check

Windows: Network connections, ipconfig

Linux: ifconfig

Use static IP addresses during troubleshooting


44/79


Step 2: Check the vNIC connection

Check if the vNIC is connected to the correct portgroup

Use VI or look into the .vmxfile

Make sure the Connected box is checked

VI: Virtual Machine Properties

The vNICconnects to this

portgroup

Is the vNICconnected


45/79


Step 2: Check the vNIC connection

Check connectivity between VMs on the same portgroup

At this point you should be able to communicate with another VM on thesame portgroup

If not

Look at the receive and transmit byte counters in the VMs to see what isgoing on

Look at esxtop, esxcfg-info for any dropped packets on the ports

Check firewall settings in the guest


46/79


47/79


Step 4: Check VLAN Configuration

Check who is tagging and stripping the VLAN IDs?

External Switch Tagging

Only the physical switch tags and strips VLAN IDs

Virtual Switch Tagging

Only the vSwitch tags and strips VLAN IDsVirtual Guest Tagging

Only the guest tags and strips VLAN IDs


48/79


Step 4: Check VLAN Configuration: EST

VLAN tagging and

stripping is done by thephysical switch

Make sure the vSwitch isnot configured to tag orstrip VLAN IDs

Check your physicalnetwork configuration

Untaggedframes

Physical switch is

responsible for thetagging and stripping

Hardware

ESX

Server

VSwitch

VMKernel

PhysicalSwitch

VSwitch


49/79


Hardware

VMKernel

VSwitch

VLAN 105 VLAN 106 VLAN 107

Physical Switch

Step 4: Check VLAN Configuration: VST

Check the portgroup VLAN

IDCheck the physical switchport configuration

Physical switch port shouldbe configured as a trunk port

Trunking should be static andunconditional

No Dynamic TrunkingProtocol (DTP)

Physical switch port trunkencapsulation should be set

to 802.1qNo ISL, LANE etc

VM on VLAN 106VM on VLAN 105 VM on VLAN 107

The switch portsees packetswith multiple

VLAN IDs

Make sure theportgroup VLANIDs are correct

vSwitch supportsonly 802.1q

encapsulation


50/79


51/79


Step 4: Check VLAN Configuration: VGT

Check if the portgroup VLAN Id is

set to 4095

Check physical switchconfiguration

Physical switch port should be astatically trunked

Physical switch should be configuredto expect frames with the specificVLAN IDs on the port

Physical switch port trunkencapsulation should be set to

802.1q

Hardware

ESX

Server

VSwitch

VMKernel

PhysicalSwitch

PortgroupVLAN ID set

to 4095


52/79


Step 4: Check VLAN Configuration: Native VLAN

Dont use native VLAN for regular

traffic

Default native VLAN is often VLAN 1

If you have to use default nativeVLAN for regular data traffic, do

one of the following:Change the native VLAN on thephysical switch

Force tagging of native VLAN frames

Might need to change native

VLAN behavior on allneighboring switches Machine withVLAN ID 1

VLAN 1 Framesnot tagged

VMconnected toa portgroupwith VLAN

ID 1

Hardware

ESX

Server

VMKernel

Physical Switch with

Native VLAN ID 1

VSwitch

vSwitch wontdeliver

untaggedpackets to the

VM


53/79


Problem

Someof the VMs on a vSwitch have network

connectivity, other dont


54/79


Step 1: Round up the Usual Suspects

Check the vNIC on the VM

Check if the vNIC is connected to the correct portgroup

Check if VM to VM traffic on the same portgroup works

Check if the physical NIC is connected to the rightport/switch

Use CDP


55/79


56/79


Step 2: NIC Teaming

VI

esxcfg-info

Search forNetwork Hint in

the output


57/79


Step 3: VLAN Configuration

The two VMs could beusing different uplinks

VLAN configuration onphysical switch portsconnected to NICs in ateam should beidentical

ESX

Serv

er

VSwitch

Physical Switch

VLAN configuration for thesephysical switch ports should

be identical


58/79


Step 3: VLAN Configuration

Manually configure theswitch port to expect allthe VLAN IDs in use

Hardware

VMKernel

VSwitch

VLAN 105 VLAN 106 VLAN 107

Physical Switch

Configure thephysical switch

port to trunkVLAN IDs 105,106 and 107


59/79


Problem

VMs have intermittent network connection


60/79



Check the vNIC on the VM

Check if VM to VM traffic on the same portgroup works withoutintermittent problems

Check VLAN configuration

Identical VLAN configuration on physical switch ports that are ina team

Make sure the NICs in a team are in the same layer 2

broadcast domainCheck if the physical NIC is connected to the right port/switch


61/79


Step 2: NIC Teaming

Port Id or MAC basedload balancing on ESX

Dont enable LinkAggregation on thephysical switch

Hardware

VMKernel

VSwitch

Physical Switch

VM AMAC A

MAC A

vSwitch expectspackets for VM Aonlyon this uplink


62/79


Step 2: NIC Teaming

IP based load balancingon ESX

Enable Link Aggregation

on the physical switchStatic Link Aggregation

No LACP or PAgP

Hardware

VMKernel

VSwitch

Physical Switch

VM AMAC A

Rx packets cancome from any

uplink

MAC A MAC A MAC A

Configure LinkAggregation on

the physicalswitch ports


63/79


Step 2: NIC Teaming

Active-Standby wont work

with IP based loadbalancing

Because of the static LinkAggregation the physicalswitch will want to deliver

packets on the standby NIC

Be careful whenconfiguring IP Hash basedteaming with other load-balancing configurations

on portgroups of the same

vSwitch

Hardware

VMKernel

VSwitch

Physical Switch

VM AMAC A

Active Standby


64/79


Step 2: NIC Teaming

Multicast traffic?

Dont use MAC Address Based Load balancing

Use Port Id based load balancing instead


65/79


66/79


Step 3: Jumbo Frames

MTU should be the sameend to end

Set vNIC MTU in the guest

Use esxcfg-vswitch to setthe MTU of the physical NIC

esxcfg-vswitch m

Use RCLI for ESXi 3.5VMKernel

MTU shouldbe

consistent


67/79


Problem

VMs lose network connectivity upon teaming

failover/failback


68/79



Check physical switch side VLAN configurations

Should be identical for all the NICs in a team

Check physical NIC connections

NICs in a team must be in the same broadcast domain


69/79


Step 2: Spanning Tree Protocol

The switch dropspackets on a newlyactive port till the port isin forwarding state, ifSTP is enabled

This interferes withfailbacks

PhysicalSwitch

Listening

Blocking

Learning

Forwarding

The switch isdropping

packets onthe port tillthe port is inForwardingState

STP States of a newlyactive port


70/79



Loops are not possible

inside ESXVSwitchVSwitch

vSwitchescannot beconnectedinternally

Packetscoming up oneuplink are not

transmitted outanother

Physical Switch


71/79



To avoid the dropped packets, do one of the following

Enable PortFast mode for the physical switch ports feeding the ESXServer

Configure the physical switch ports feeding the ESX Server as EdgePorts when using Rapid Spanning Tree Protocol

Disable STP for the physical switch ports feeding the ESX Server

This is not a recommendation to disable STP in the entirenetwork


72/79


Problem

VMs lose network connectivity after VMotion


73/79


Step 1: Basics

Hardware

ESX

Server

VSwitch

VMKernel

Hardware

ESX

Server

VSwitch

VMKernel

Physical Switch Physical Switch

These NICs shouldbe in the same

broadcast domain

These physicalswitch ports should

have identicalVLAN configuration

VMotion


74/79


Step 2: Notify Switch

Hardware

ESX

Server

VSwitch

VMKernel

Hardware

ESX

Server

VMKernel

VMotion

MAC A

Physical Switch

MAC A MAC B

Physical Switch

MAC C

MAC BMAC B

MAC B

MAC B MAC C

VSwitch

RARP Packet

MAC B


75/79


Notify Switch

Notify switch is enabledby default

Settings should reflect

application requirements


76/79


77/79


Step 1: Check Portgroup Security Policies

Promiscuous ModeIf allowed, guest receives all frames on the vSwitch

Some applications need promiscuous mode

Network sniffers

Intrusion detection systems

MAC Address ChangeIf allowed, guest can change its MAC address

Implication: Malicious guests can spoof MAC addresses

Forged TransmitsIf allowed, guest can send packets with different source MAC

Implication: Malicious guests can spoof MAC addresses or causeMAC Flooding

Security settings should reflect applicationrequirements

Example: Microsoft Network Load Balancing


78/79


Example: Microsoft Network Load BalancingIn Unicast Mode

All cluster hosts are assigned the same MAC address

Thus incoming packets are received by all cluster hosts

Uses forged MAC addresses to hide the cluster MAC addressfrom the switch

Prevents the switch from learning the cluster's actual MAC address

Incoming packets for the cluster are delivered to all switch ports

Portgroup configurations

Allow MAC address changes

Allow Forged Transmits

Do not Notify Switch

KB Article 1556http://kb.vmware.com/kb/1556

Recommendation: Use NLB in Multicast Mode


79/79

For more information:

VMware Networking Technologyvmware.com/go/networking

VMware Networking Blogblogs.vmware.com/networking
http://vmware.com/go/networkinghttp://blogs.vmware.com/networkinghttp://blogs.vmware.com/networkinghttp://vmware.com/go/networking

Vi Networking Adv Troubleshooting

Documents

Transcript of Vi Networking Adv Troubleshooting