VI3 Networking: Advanced Configuration and Troubleshooting

vmware.com/go/networking

VI3 Networking:Advanced Troubleshooting

22 vmware.com/go/networkingvmware.com/go/networking

ESX Networking Architecture

Physical NICs

Virtual Switch

Virtual NICs

VM’s Virtual NIC

Flexible

Enhanced

Virtual E1000

Vswif for the Service

Console

VMkernel uses vmknic

VMkernel TCP/IP Stack

Physical

Switches

Hardware

ESX Server

VMKernel NIC VSwitch

VMKernel

VMotion iSCSINFS

VMKernel TCP/IP Stack

Agenda

Basic Troubleshooting

How to isolate problems?

What tools are available for troubleshooting?

Troubleshooting Scenarios

Step-by-step guide on how to troubleshoot some specific

networking problems

Basic Troubleshooting Techniques

Isolate the problem

Troubleshoot one component at a time

Physical NICs

Virtual Switch

Virtual NICs

Physical Network

Tools for troubleshooting

Command Line Utilities

Third party tools

Ping and traceroute

Traffic sniffers and Protocol Analyzers

Wireshark

Hardware

VMKernel

VSwitch

ESX Server

Isolating Network Problems: Physical NICs

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

What to look for?

Where to look?

Physical NICs: What to look for?

Does the device show up?

Is the driver loaded?

Physical properties of the link

Link State

Link Speed

Duplex Setting

MTU settings

Is the NIC connected to where you want it to be connected?

Is the NIC working?

Is the NIC transmitting and receiving packets?

Is the NIC dropping any packets?

esxcfg-nics

ifconfig inside Service Console

VI Client

Network Hints

Cisco Discovery Protocol (CDP)

ifconfig inside Service Console

esxtop/resxtop

esxcfg-info

Getting information about the physical NICs

VI Client provides basic information about the physical

Type of NIC Link Status Connections Network Hint

esxcfg-nics allows you to set or get physical NIC

settings via the command line

The ‘- l’ option lists the nics in the system and their settings

Link State

Duplex

Where is the physical NIC connected to?

Just follow the cable

Use CDP and Network Hints

Cisco Discovery Protocol

Periodic exchange of

information

Physical switch port a vmnic is

connected to

vSwitch a physical switch port is

connected to

Duplex and speed settings

Cisco Discovery Protocol

CDP is enabled by default in listening mode

On ESX Server 3.5, it is possible to configure CDP also inadvertising mode

Enabled/disabled only via command line with

esxcfg-vswitch –B <state> <vSwitch>

States

Listen

Advertise

Verify the setting with

esxcfg-vswitch –b <vSwitch>

esxtop provides system-wide real-time traffic information

For ESXi use resxtop utility provided in the RCLI

Type ‘n’ to switch to the network utilization screen

Output of esxtop

On ESX Server 3.5, running ifconfig inside Service Console provides information valuable for troubleshooting

Output of ifconfig

esxcfg-info provides detailed information about the

system

Use ‘esxcfg-info –n’ for network information

Redirect the output of esxcfg-info to a file

Look for the ‘Physical Nic’ section

Output of esxcfg-info

Search for vmnicX in the

output of esxcfg-info

Summary: Getting information about the physical NICs

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

VI Client

esxcfg-nics

esxtop/resxtop

ifconfig

esxcfg-info

Isolating Network Problems: Virtual Switch

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

Virtual Switch: What to look for?

vSwitch and Portgroup Configuration

Uplinks

VLAN Setting

Layer 2 Security Policies

NIC Teaming Configuration

Is the traffic flowing through the

vSwitch?

Is the vSwitch dropping any packets?

esxcfg-vswitch

esxcfg-info

VI Client

esxtop/resxtop

esxcfg-info

Getting information about the vSwitch: VI

VI: Virtual Switch

Configuration

VI: Portgroup

Properties

Getting information about the vSwitch: esxcfg-vswitch

esxcfg-vswitch

An interface for adding, removing, and modifying virtual switches

and their settings

Output of esxcfg-vswitch -l

Getting information about the vSwitch: esxtop

esxtop provides system-wide real-time traffic information

For ESXi use resxtop utility provided in the RCLI

Type ‘n’ to switch to the network utilization screen

Real Time Traffic

Information

Output of esxtop

Getting information about the vSwitch: esxtop

System Running Flood

Ping to the vmknic

Hardware

Server

VMKernel

Service

Console

VMkernel

Physical

Switch

vmnic3

vmnic4

vmknic traffic is going

through vmnic4

Output of esxtop

Getting information about the vSwitch: esxcfg-info

esxcfg-info provides information in greater detail

Configuration information

Getting information about the vSwitch: esxcfg-info

esxcfg-info provides information in greater detail

Cumulative traffic information for each port on the vSwitch

Information about VMkernel

Information about Uplink Port

(vmnic4)

Summary: Getting information about the vSwitch

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

VI Client

esxtop/resxtop

esxcfg-vswitch

esxcfg-info

Isolating Network Problems: Virtual NICs

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

Virtual NICs: What to look for?

Does the device show up?

Is the driver loaded?

Physical properties of the link

Link State

MTU settings

Is the vNIC connected to the correct portgroup?

Portgroup using the correct uplink

Portgroup with the correct security properties

Is the NIC working?

Does the NIC have an IP address?

Is the NIC transmitting and receiving packets?

Is the NIC dropping any packets?

esxcfg-vswif

esxcfg-vmknic

Guest specific utilities

ifconfig

Windows

Device Manager

VI Client

.vmx file

esxcfg-info

Guest specific utilities

ifconfig

Windows

Network Connections

esxtop/resxtop

esxcfg-info

Getting information about the vNIC

VI: Virtual Machine

Properties

.vmx file

VM’s Connection Information

esxcfg-vswif

An interface to configure Service Console NIC

esxcfg-vmknic

An interface to configure VMkernel NIC

Output of esxcfg-vswif -l

Output of esxcfg-vmknic -l

Output of esxtop

Search for the port

ID of the vNIC in

the esxcfg-info

output

Look for Rx/Tx

information for the

vNIC you are

interested in

Cumulative Traffic

Information

Real time traffic

information

Summary: Getting information about the vNIC

Physical

Switches

Hardware

ESX Server

VMKernel

VMotion iSCSINFS

VI Client

Guest Utilities

esxtop/resxtop

esxcfg-info

Sniffing For Trouble

Sniff for packets at different layers for isolation

Physical Switch Port Level

vSwitch Level

VM Level

Look for

Lost Packets

Large number of packet retransmissions

Anomalies reported by protocol analyzers like Wireshark etc.

Look for patterns

Are packets of a certain type causing problems?

Are packets of a certain size causing problems?

Hardware

Server

VMKernel

Physical

Switch

VSwitch

Mirrored Port

Capture packet

traces inside

the VM

Capture packet

traces on the

vSwitch

Collecting Network Traces on the vSwitch

Hardware

Server

VMKernel

VSwitch

Set the VLAN ID of the

Service Console portgroup

to 4095

Enable promiscuous mode

for the Service Console

portgroup

VM A on VLAN 106

Run tcpdump –i vswifX

in the Service Console Running

tcpdump –i

vmnic0 won’t

VLAN 106 Packet For VM A

VLAN 106 Packet For VM A VLAN 106 Packet For VM A

Collecting Network Traces on the vSwitch

Hardware

Server

VMKernel

VSwitch

VM A on VLAN 106

Create a portgroup

Set the VLAN ID of the

portgroup to 4095

Enable promiscuous mode

for the portgroup

Run Wireshark in the

VLAN 106 Packet For VM A

VLAN 106 Packet For VM A VLAN 106 Packet For VM A

Logs on ESX

VMkernel logs

/var/log/vmkernel for ESX

/var/log/messages for ESXi

VM logs

vmware.log file in the VM

directory

Service console logs

/var/log/messages for ESX

Also check the guest OS

logs for any errors

Hardware

Server

VMKernel

VSwitch

VMkernel Logs

VM Logs

Guest OS Logs

Service Console Logs

Troubleshooting Scenarios

Signs of trouble

Basic connectivity problems

No network connectivity on some or all of the VMs on a vSwitch

Flaky network connection

Connection timeouts

Intermittent loss of connectivity

Problem

None of the VMs on my ESX box have network

connectivity

Step 1: Check inside the VM

Has the guest OS detected the vNIC?

Use OS specific utilities to check

Windows: Device Manager

Linux: lspci, ifconfig

If a network interface for the vNIC does not show up

Check the VM configuration using VI or by looking into the .vmx file

Check the VM’s log file (vmware.log) for any obvious problems

Check the guest OS log files for any obvious problems

Is appropriate driver for the vNIC installed and loaded?

Install VMware tools inside the guest

Step 1: Check inside the VM

Is the network interface in the guest up and does it have

an IP address?

Use OS specific utilities to check

Windows: Network connections, ipconfig

Linux: ifconfig

Use static IP addresses during troubleshooting

Step 2: Check the vNIC connection

Check if the vNIC is connected to the correct portgroup

Use VI or look into the .vmx file

Make sure the ‘Connected’ box is checked

VI: Virtual Machine Properties

The vNIC

connects to this

portgroup

Is the vNIC

connected

Step 2: Check the vNIC connection

Check connectivity between VMs on the same portgroup

At this point you should be able to communicate with another VM on the same portgroup

If not

Look at the receive and transmit byte counters in the VMs to see what is going on

Look at esxtop, esxcfg-info for any dropped packets on the ports

Check firewall settings in the guest

Step 3: Check the physical NIC

Check if the uplinks connected to vSwitch and the portgroup are correct

Use VI or esxcfg-vswitch -l

Check if the physical NIC is receiving and transmitting data

Check the RX and TX counter using ifconfig, esxcfg-info or esxtop

Check if Tx/Rx error counters are incrementing

Check the VMkernel logs for any errors

Check if the physical NICs are connected to the correct physical switch ports

Use CDP or Network Hints

Step 4: Check VLAN Configuration

Check who is tagging and stripping the VLAN IDs?

External Switch Tagging

Only the physical switch tags and strips VLAN IDs

Virtual Switch Tagging

Only the vSwitch tags and strips VLAN IDs

Virtual Guest Tagging

Only the guest tags and strips VLAN IDs

Step 4: Check VLAN Configuration: EST

VLAN tagging and stripping is done by the physical switch

Make sure the vSwitch is not configured to tag or strip VLAN IDs

Check your physical network configuration

Untagged

frames

Physical switch is

responsible for the

tagging and stripping

Hardware

Server

VSwitch

VMKernel

Physical

Switch

VSwitch

Hardware

VMKernel

VSwitch

VLAN 105 VLAN 106 VLAN 107

Physical Switch

Step 4: Check VLAN Configuration: VST

Check the portgroup VLAN ID

Check the physical switch port configuration

Physical switch port should be configured as a trunk port

Trunking should be static and unconditional

No Dynamic Trunking Protocol (DTP)

Physical switch port trunk encapsulation should be set to 802.1q

No ISL, LANE etc

VM on VLAN 106VM on VLAN 105 VM on VLAN 107

The switch port

sees packets

with multiple

VLAN IDs

Make sure the

portgroup VLAN

IDs are correct

vSwitch supports

only 802.1q

encapsulation

Hardware

VMKernel

VSwitch

Physical Switch

Step 4: Check VLAN Configuration: VST

Check if the physical switch is configured to trunk all the VLANs on the vSwitch

Manually specify all the VLANs to be trunked

No VLAN Trunking Protocol (VTP)

VM on VLAN 106VM on VLAN 105 VM on VLAN 107

Configure the

physical switch

to trunk VLAN

IDs 105, 106 and

Step 4: Check VLAN Configuration: VGT

Check if the portgroup VLAN Id is set to 4095

Check physical switch configuration

Physical switch port should be a statically trunked

Physical switch should be configured to expect frames with the specific VLAN IDs on the port

Physical switch port trunk encapsulation should be set to 802.1q

Hardware

Server

VSwitch

VMKernel

Physical

Switch

Portgroup

VLAN ID set

to 4095

Step 4: Check VLAN Configuration: Native VLAN

Don’t use native VLAN for regular

traffic

Default native VLAN is often VLAN 1

If you have to use default native

VLAN for regular data traffic, do

one of the following:

Change the native VLAN on the

physical switch

Force tagging of native VLAN frames

Might need to change native

VLAN behavior on all

neighboring switches Machine with

VLAN ID 1

VLAN 1 Frames

not tagged

connected to

a portgroup

with VLAN

Hardware

Server

VMKernel

Physical Switch with

Native VLAN ID 1

VSwitch

vSwitch wont

deliver

untagged

packets to the

Problem

Some of the VMs on a vSwitch have network

connectivity, other don’t

Step 1: Round up the Usual Suspects

Check the vNIC on the VM

Check if the vNIC is connected to the correct portgroup

Check if VM to VM traffic on the same portgroup works

Check if the physical NIC is connected to the right

port/switch

Use CDP

Step 2: NIC Teaming

The two VMs could be

using different uplinks

NICs in a team should

be connected to the

same broadcast

domain

Look at the Network Hint

Server

VSwitch

Physical Switch

These two NICs

should be in the same

broadcast domain

Step 2: NIC Teaming

esxcfg-info

Search for

‘Network Hint’ in

the output

Step 3: VLAN Configuration

The two VMs could be

using different uplinks

VLAN configuration on

physical switch ports

connected to NICs in a

team should be

identical

Server

VSwitch

Physical Switch

VLAN configuration for these

physical switch ports should

be identical

Step 3: VLAN Configuration

Manually configure the

switch port to expect all

the VLAN IDs in use

Hardware

VMKernel

VSwitch

Physical Switch

Configure the

physical switch

port to trunk

VLAN IDs 105,

106 and 107

Problem

VMs have intermittent network connection

Check the vNIC on the VM

Check if VM to VM traffic on the same portgroup works without intermittent problems

Check VLAN configuration

Identical VLAN configuration on physical switch ports that are in a team

Make sure the NICs in a team are in the same layer 2 broadcast domain

Check if the physical NIC is connected to the right port/switch

Step 2: NIC Teaming

Port Id or MAC based load balancing on ESX

Don’t enable Link Aggregation on the

physical switch

Hardware

VMKernel

VSwitch

Physical Switch

vSwitch expects

packets for VM A

only on this uplink

Step 2: NIC Teaming

IP based load balancing on ESX

Enable Link Aggregation on the physical switch

Static Link Aggregation

No LACP or PAgP

Hardware

VMKernel

VSwitch

Physical Switch

Rx packets can

come from any

uplink

MAC A MAC A MAC A

Configure Link

Aggregation on

the physical

switch ports

Step 2: NIC Teaming

Active-Standby won’t work

with IP based load

balancing

Because of the static Link

Aggregation the physical

switch will want to deliver

packets on the standby NIC

Be careful when

configuring IP Hash based

teaming with other load-

balancing configurations

on portgroups of the same

vSwitch

Hardware

VMKernel

VSwitch

Physical Switch

Active Standby

Step 2: NIC Teaming

Multicast traffic?

Don’t use MAC Address Based Load balancing

Use Port Id based load balancing instead

Step 3: Jumbo Frames

Network devices drop

frames larger than MTU

How to detect such

problems?

Run ping with large packet

size, e.g 20000

ping -s <packet size>

Check for packet drops on the

vNICs and the physical NICsPhysical Switch

MTU = 9000

MTU = 1500

This interface

will drop

packets larger

than 1500

Step 3: Jumbo Frames

MTU should be the same

end to end

Set vNIC MTU in the guest

Use esxcfg-vswitch to set

the MTU of the physical NIC

esxcfg-vswitch –m <MTU> <vSwitch>

Use RCLI for ESXi 3.5VMKernel

MTU should

consistent

Problem

VMs lose network connectivity upon teaming

failover/failback

Check physical switch side VLAN configurations

Should be identical for all the NICs in a team

Check physical NIC connections

NICs in a team must be in the same broadcast domain

Step 2: Spanning Tree Protocol

The switch drops

packets on a newly

active port till the port is

in forwarding state, if

STP is enabled

This interferes with

failbacks

Physical

Switch

Listening

Blocking

Learning

Forwarding

The switch is

dropping

packets on

the port till

the port is in

Forwarding

STP States of a newly

active port

Loops are not possible

inside ESX VSwitchVSwitch

vSwitches

cannot be

connected

internally

Packets

coming up one

uplink are not

transmitted out

another

Physical Switch

To avoid the dropped packets, do one of the following

Enable PortFast mode for the physical switch ports feeding the ESX

Server

Configure the physical switch ports feeding the ESX Server as Edge

Ports when using Rapid Spanning Tree Protocol

Disable STP for the physical switch ports feeding the ESX Server

This is not a recommendation to disable STP in the entire

network

Problem

VMs lose network connectivity after VMotion

Step 1: Basics

Hardware

Server

VSwitch

VMKernel

Hardware

Server

VSwitch

VMKernel

Physical Switch Physical Switch

These NICs should

be in the same

broadcast domain

These physical

switch ports should

have identical

VLAN configuration

VMotion

Step 2: Notify Switch

Hardware

Server

VSwitch

VMKernel

Hardware

Server

VMKernel

VMotion

Physical Switch

MAC A MAC B

Physical Switch

MAC BMAC B

MAC B MAC C

VSwitch

RARP Packet

Notify Switch

Notify switch is enabled by default

Settings should reflect application requirements

Problem

Network connectivity is fine on the VMs but my

application does not work

Step 1: Check Portgroup Security Policies

Promiscuous Mode

If allowed, guest receives all frames on the vSwitch

Some applications need promiscuous mode

Network sniffers

Intrusion detection systems

MAC Address Change

If allowed, guest can change its MAC address

Implication: Malicious guests can spoof MAC addresses

Forged TransmitsIf allowed, guest can send packets with different source MAC

Implication: Malicious guests can spoof MAC addresses or cause MAC Flooding

Security settings should reflect application requirements

Example: Microsoft Network Load Balancing In Unicast Mode

All cluster hosts are assigned the same MAC address

Thus incoming packets are received by all cluster hosts

Uses forged MAC addresses to hide the cluster MAC address from the switch

Prevents the switch from learning the cluster's actual MAC address

Incoming packets for the cluster are delivered to all switch ports

Portgroup configurations

Allow MAC address changes

Allow Forged Transmits

Do not Notify Switch

KB Article 1556

http://kb.vmware.com/kb/1556

Recommendation: Use NLB in Multicast Mode

For more information:

VMware Networking Technologyvmware.com/go/networking

VMware Networking Blogblogs.vmware.com/networking

VI3 Networking: Advanced Configuration and Troubleshooting

Documents

Transcript of VI3 Networking: Advanced Configuration and Troubleshooting

Vi3!35!25 Resource Mgmt

Assembling, Troubleshooting and Networking

Vi3!35!25 u2 Admin Guide

Vi3-Leaflet-ENG-011019 · Title: Vi3-Leaflet-ENG-011019 Created Date: 10/1/2019 6:19:12 PM

MELJUN CORTES PC assembly-troubleshooting-networking

VI3 Overview

VI3 IC REV B - 03 Networking

Firewall Typical Networking and Troubleshooting Common Faults

Troubleshooting Tips for Data communications, Networking & TCP/IP

Topics Networking Concepts Storage Concepts Presentation by Anand Mewalal Vmware VI3 Concepts On networking and Storage.

Vi3!35!25 Admin Guide

9: Troubleshooting Your Network Networking for Home & Small Business.

Firewall Typical Networking and Troubleshooting Common Faults.

Vi3 Vum 10 Admin Guide

Vi3 Admin Guide

Chapter12 -- troubleshooting networking problems

Vi3 301 201 Server Config

Vi3 Vec 10 Admin Guide

Vi3 San Design Deploy

Presentation on CCNA(Networking & Troubleshooting)