Best Practices for Virtual Networking Karim Elatov ... · > Uplink; No vSwitch ... Sample...
Transcript of Best Practices for Virtual Networking Karim Elatov ... · > Uplink; No vSwitch ... Sample...
© 2009 VMware Inc. All rights reserved
Best Practices for Virtual Networking
Karim Elatov
Technical Support Engineer, GSS
2
Agenda
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
3
Virtu
al S
witc
h
Virtual
Conventional access, distribution, core design
Design with redundancy for enhanced availability
Under the covers, virtual network same as physical
Access layer implemented as virtual switches
Ph
ys
ica
l
Sw
itch
Ph
ys
ica
l
Sw
itch
Physical
Virtual Network Overview - Physical to Virtual
Physical
4
vNetwork Distributed
Switch
Distributed:
1 or more per
“Datacenter”
- Expanded feature set
- Private VLANs
- Bi-directional traffic shaping
- Network vMotion
- Simplified management
Virtual networking concepts similar with all virtual switches
Virtual Switch Options
Virtual Switch Model Details
vNetwork Standard
Switch
Host based:
1 or more per
ESX host
- Same as vSwitch in VI3
Cisco Nexus 1000V Distributed:
1 or more per
“Datacenter”
- Cisco Catalyst/Nexus feature set
- Cisco NXOS cli
- Supports LACP
5
ESX Virtual Switch: Capabilities
Layer 2 - only forward frames VM <-> VM and VM <-
> Uplink; No vSwitch <-> vSwitch or Uplink <-> Uplink
MAC
address
assigned to
vnic
VM0 VM1
vSwitch
Physical
Switches
vSwitch
MAC a MAC b MAC c
vSwitch will not create loops affecting Spanning
Tree in the physical network
Can terminate VLAN trunks (VST mode) or pass
trunk through to VM (VGT mode)
NIC Teaming of Physical NIC(s) [uplink(s)] associated
with vSwitches
6
Distributed Virtual Switch
vCenter vCenter
Standard vSwitch vNetwork & dvSwitch
Exist across 2 or more clustered hosts
•Provide similar functionality to vSwitches
•Reside on top of hidden vSwitches
vCenter owns the configuration of the dvSwitch
•Consistent host network configurations
7
Port Groups
Template for one or more ports with a common
configuration
• VLAN Assignment
• Security
• Traffic Shaping (limit egress traffic from VM)
• Failover & Load Balancing
Distributed Virtual Port Group (Distributed Virtual Switch)
• Bidirectional traffic shaping (ingress and egress)
• Network VMotion—network port state migrated upon
VMotion
8
NIC Teaming for Availability and Load Sharing
NIC Teaming aggregates multiple physical uplinks:
• Availability—reduce exposure to single points of
failure (NIC, uplink, physical switch)
• Load Sharing—distribute load over multiple
uplinks (according to selected NIC teaming
algorithm)
VM0 VM1
vSwitch
NIC Team
KB - NIC teaming in ESXi and ESX (1004088)
Requirements:
• Two or more NICs on same vSwitch
• Teamed NICs must have same VLAN configurations
9
NIC Teaming Options
Explicit Failover
Order
Highest order uplink
from active list
Teamed ports in same L2 domain
(BP: team over two physical
switches)
Best Practices:
•Originating Virtual PortID for VMs is the default, no extra configuration needed
•IP Hash, ensure that physical switch is properly configured for Etherchannel
*KB - ESX/ESXi host requirements for link aggregation (1001938)
*KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048)
Name Algorithm—vmnic
chosen based upon:
Physical Network Considerations
Originating
Virtual Port ID
vnic port Teamed ports in same L2 domain
(BP: team over two physical
switches)
Source MAC
Address
MAC seen on vnic Teamed ports in same L2 domain
(BP: team over two physical
switches)
IP Hash* Hash(SrcIP, DstIP) Teamed ports configured in static
802.3ad “Etherchannel”
- no LACP (Nexus 1000v for LACP)
- Needs MEC to span 2 switches
10
Cisco Nexus 1000v Overview
Cisco Nexus 1000v is a software switch for vNetwork Distributed
Switches (vDS):
• Virtual Supervisor Module (VSM)
• Virtual Ethernet Module (VEM)
Things to remember:
• Virtual Ethernet Module (VEM)VSM uses external network fabric to
communicate with VEMs
• VSM does not take part in forwarding packets
• VEM does not switch traffic to other VEM without an uplink
11
Cisco Nexus 1000v Modules
vCenter Server
VMware ESX
Server 1
VMware vSwitch
VMware ESX
Server 2
VMware vSwitch
VMware ESX
Server 3
VMware vSwitch
VM #1
VM #4
VM #3
VM #2
VM #5
VM #8
VM #7
VM #6
VM #9
VM #12
VM #11
VM #10
Nexus 1000V
VSM
VEM VEM VEM Nexus 1000V vDS
Virtual Supervisor Module (VSM)
• Virtual or Physical appliance running Cisco OS (supports HA)
• Performs management, monitoring, & configuration
• Tight integration with VMware Virtual Center
Virtual Ethernet Module (VEM)
• Enables advanced networking capability on the hypervisor
• Provides each VM with dedicated “switch port”
• Collection of VEMs = 1 DVS
Cisco Nexus 1000V Enables:
• Policy Based VM Connectivity
• Mobility of Network & Security Properties
• Non-Disruptive Operational Model
12
vSwitch Configurations
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
13
Cisco ‘show run’ and ‘show tech-support’
The following is a Cisco EtherChannel sample configuration: interface Port-channel1 switchport switchport access vlan 100 switchport mode access no ip address ! interface GigabitEthernet1/1 switchport switchport access vlan 100 switchport mode access no ip address channel-group 1 mode on !
Obtain configuration of a Cisco router or switch
•Run commands in priviliged EXEC mode
•’show run’
•‘show tech-support’
KB - Troubleshooting network issues with the Cisco show tech-support command (1015437)
14
Traffic Types on a Virtual Network
Virtual Machine Traffic
• Traffic sourced and received from virtual machine(s)
• Isolate from each other based on service level
How do we maintain traffic isolation without proliferating NICs? VLANs
vMotion Traffic
• Traffic sent when moving a virtual machine from one ESX host to another
• Should be isolated
Management Traffic
• Should be isolated from VM traffic (one or two Service Consoles)
• If VMware HA is enabled, includes heartbeats
IP Storage Traffic—NFS and/or iSCSI via vmkernel interface
• Should be isolated from other traffic types
Fault Tolerance (FT) Logging Traffic
• Low latency, high bandwidth
• Should be isolated from other traffic types
15
Traffic Types on a Virtual Network, cont.
Port groups in dedicated VLANs on a management-only virtual
switch.
vMotion
production
virtual switch
Service console/VMK Interface
virtual machines vMotion
106
storage
107
mgmt
108 management
virtual switch
production management storage
16
VLAN Tagging Options
vSwitch
Physical Switch
VST – Virtual Switch Tagging
VLAN Tags
applied in
vSwitch
VST is the best practice and
most common method
VLAN
assigned in
Port Group
policy
vSwitch
Physical Switch
EST – External Switch Tagging
External Physical
switch applies
VLAN tags
switchport access vlan switchport trunk
vSwitch
Physical Switch
VGT – Virtual Guest Tagging
VLAN Tags
applied in
Guest
PortGroup
set to VLAN
“4095”
switchport trunk
17
DVS Support for Private VLAN (PVLAN)
Enable users to restrict communications
• Between VMs on the same VLAN or network segment
PVLAN Types
• Community
• VMs can communicate with VMs on Community and Promiscuous
• Isolated
• VMs can only communicate with VMs on the Promiscuous
• Promiscuous
• VMs can communicate with all VMs
KB - Private VLAN (PVLAN) on vNetwork Distributed Switch - Concept Overview (1010691)
Allow devices to share the same IP subnet while being Layer 2 Isolated
Benefits:
•Employ Larger subnets (advantageous to hosting environments)
•Reduce Management Overhead
application
server
Web
server database
server
server document
server
isolated
PVLAN
isolated
PVLAN community PVLAN
DMZ network
router in promiscuous PVLAN
18
W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B
Distributed Virtual Switch
PG PG PG PG PG PG PG PG PG PG PG PG
TOTAL COST: 12 VLANs (one per VM)
TOTAL COST: 1 PVLAN (over 90% savings…)
W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B W2003EE-32-A W2003EE-32-B
Distributed Virtual Switch
PG (with Isolated PVLAN)
PVLAN Cost Benefit
19
Link Aggregation
EtherChannel
•Port trunking between two to eight
•Active Fast Ethernet, Gigabit Ethernet, or 10 Gigabit Ethernet ports
KB ESX/ESXi host requirements for link aggregation (1001938)
LACP (one of the implementations included in IEEE 802.3ad)
•Link Aggregation Control Protocol (LACP)
•Control the bundling of several physical ports into a single logical channel
•Only supported on Nexus 1000v
EtherChannel vs. 802.3ad
•EtherChannel is Cisco proprietary and 802.3ad is an open standard
Note: ESX implements 802.3ad Static Mode Link Aggregation
20
Sample Link Aggregation Configuration
KB - Sample configuration of EtherChannel / Link aggregation with ESX/ESXi andCisco/HP switches (1004048)
Supported switch Aggregation algorithm: IP-SRC-DST Supported Virtual Switch NIC Teaming mode: IP HASH
21
Failover Configurations
Figure — Using beacons to detect upstream
network connection failures. KB - What is beacon probing? (1005577)
Beacon Probing sends out and listens for beacon probes
•Broadcast frames (ethertype 0x05ff)
Beacon Probing Best Practice
•Use at least 3 NICs for triangulation
•If only 2 NICs in team, can’t determine link failed
•Leads to shotgun mode results
Link Status relies solely on the network adapter link state
•Cannot detect configuration errors
•Spanning Tree Blocking
•Incorrect VLAN
•Physical switch cable pulls
22
Spanning Tree Protocol (STP) Considerations
Spanning Tree Protocol creates loop-free L2 tree topologies in the physical network
• Physical links put in “blocking” state to construct loop-free tree
VM0 VM1
vSwitch
Physical
Switches
MAC a MAC b
Switches sending
BPDUs every 2s to
construct and
maintain Spanning
Tree Topology
vSwitch drops
BPDUs
Blocked link
Recommendations for Physical Network Config: 1. Leave Spanning Tree enabled on physical network
and ESX facing ports (i.e. leave it as is!) 2. Use “portfast” or “portfast trunk” on ESX facing
ports (puts ports in forwarding state immediately) 3. Use “bpduguard” to enforce STP boundary
KB - STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804)
ESX vSwitch does not participate in Spanning Tree and will not create loops with uplinks
•ESX Uplinks will not block, always active (full use of all links)
23
Tips & Tricks
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
24
Tips & Tricks
Load-Based Teaming (LBT)
• Dynamically balance network load over available uplinks
• Triggered by ingress or egress congestion at 75% mean utilization over a 30
second period
• Configure on DVS via “Route based on physical NIC load”
*LBT is not available on the Standard vSwitch (DVS feature for ingress/egress traffic shaping)
Network I/O Control (NetIOC)
• DVS software scheduler to isolate and prioritize specific traffic types
contending for bandwidth on the uplinks connecting ESX/ESXi 4.1 hosts with
the physical network.
25
Tips & Tricks
Tip #5 – Link aggregation is never supported on disparate trunked switches – Use
VSS with MEC. (KB 1001938 & KB 1027731)
Tip #4 - Beacon Probing and IP Hash DO NOT MIX (duplicate packets and port flapping) (KB 1017612 & KB 1012819)
Tip #1 – After physical to virtual migration, the VM MAC address can be changed for Licensed Applications relying on physical MAC address. (KB 1008473)
Tip #2 – NLB Multicast needs physical switch Manual ARP resolution of NLB cluster. (KB 1006525)
Tip #3 – Cisco Discovery Protocol (CDP) gives switchport configuration information useful for troubleshooting (KB 1007069)
26
Using 10GigE
2x 10GigE common/expected
• 10GigE CNAs or NICs
Possible Deployment Method
• Active/Standby on all Portgroups
• VMs “sticky” to one vmnic
• SC/vmk ports sticky to other
• Use Ingress Traffic Shaping
to control traffic type per
Port Group
• If FCoE, use Priority Group
bandwidth reservation (on CNA
utility)
vSwitch
iSCSI NFS VMotion FT SC
FCoE FCoE
SC#2
FCoE
10
FCoE Priority Group
bandwidth reservation
(in CNA config utility)
Gbps 10GE 10GE
Ingress (into switch)
traffic shaping policy
control on Port Group
1-2G Low b/w High
b/w
Variable/high
b/w 2Gbps+
Tips & Tricks
Best Practice: Ensure Drivers and Firmware are compatible for success
vSphere 4.1 supports up to (4) 10GigE NICs; 5.0 supports (8) 10GigE NICs
27
Troubleshooting Virtual Networks
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
28
Network Troubleshooting Tips
Troubleshoot one component at a time
• Physical NICs
• Virtual Switch
• Virtual NICs
• Physical Network
Tools for Troubleshooting
• vSphere Client
• Command Line Utilities
• ESXTOP
• Third party tools
• Ping and Traceroute
• Traffic sniffers & Protocol Analyzers
• Wireshark
• Logs
29
Capturing Traffic
ESXi uses tcpdump-uw (KB 1031186)
vSwitch must be in Promiscuous Mode (KBs 1004099 & 1002934)
Best Practice: create a new management interface for this purpose
30
What’s New in vSphere 5.0
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
31
What’s New in vSphere 5?
Monitor and troubleshoot virtual infrastructure traffic
• NetFlow V5
• Port mirror (SPAN)
• LLDP (standard based link layer discovery protocol) support simplifies the
network configuration and management in non-Cisco switch environment.
Enhancements to the network I/O control (NIOC)
• Ability to create User-defined resource pool
• Support for vSphere replication traffic type; a new system traffic type that
carries replication traffic from one host to another.
• Support for IEEE 802.1p tagging
What’s New in VMware vSphere 5.0 Networking Technical Whitepaper
32
Network Design Considerations
Best Practices for Virtual Networking
Troubleshooting Virtual Networks
Virtual Network Overview
vSwitch Configurations
What’s New in vSphere 5.0
Tips & Tricks
Network Design Considerations
33
Network Design Considerations
How do you design the virtual network for performance and availability but maintain isolation between the various traffic types (e.g. VM traffic, VMotion, and Management)?
• 2 NIC minimum for availability, 4+ NICs
per server preferred
• 802.1Q VLAN trunking highly recommended for logical scaling
(particularly with low NIC port servers)
• Examples are meant as guidance and do not represent strict
requirements in terms of design
• Understand your requirements and resultant traffic types and
design accordingly
• Starting point depends on:
• Number of available physical ports on server
• Required traffic types
34
Candidate Design:
• Team both NIC ports
• Create one virtual switch
• Create three port groups:
• Use Active/Standby policy
for each portgroup
• Portgroup1: Service Console (SC)
• Portgroup2: VMotion
• Portgroup3: VM traffic
• Use VLAN trunking
• Trunk VLANs 10, 20,
30 on each uplink
vmnic0
Active
Standby
vmnic1
Portgroup3
VLAN 30
VLAN Trunks
(VLANs 10, 20, 30)
Note: Team over dvUplinks with vDS
Portgroup1
VLAN 10
Portgroup2
VLAN 20
SC vmkernel
vSwitch
Example 1: Blade Server with 2 NIC Ports
35
Candidate Design:
• Create two virtual switches
• Team two NICs to each vSwitch
• vSwitch0 (use active/standby
for each portgroup):
• Portgroup1: Service Console (SC)
• Portgroup2: VMotion
• vSwitch1 (use Originating Virtual
PortID)
• Portgroup3: VM traffic #1
• Portgroup4: VM traffic #2
• Use VLAN trunking
• vmnic1 and vmnic3: Trunk VLANs 10, 20
• vmnic0 and vmnic2: Trunk VLANs 30, 40
VLANs
10, 20
vSwitch0 vSwitch1
VLANs
30, 40
Note: Team over dvUplinks with vDS
Active
Standby
SC vmkernel
Portgroup1
VLAN 10
Portgroup2
VLAN 20
vmnic0 vmnic1 vmnic3
Portgroup3
VLAN 30
Portgroup4
VLAN 40
vmnic2
Example 2: Server with 4 NIC Ports
36
Candidate Design:
• Create one virtual switch
• Create two NIC teams
• vSwitch0 (use active/standby
for portgroups 1 & 2):
• Portgroup1: Service Console (SC)
• Portgroup2: Vmotion
• Use Originating Virtual PortID
for Portgroups 3 & 4
• Portgroup3: VM traffic #1
• Portgroup4: VM traffic #2
• Use VLAN trunking
• vmnic1 and vmnic3: Trunk VLANs 10, 20
• vmnic0 and vmnic2: Trunk VLANs 30, 40
vmnic0
SC vmkernel
vmnic1 vmnic2 vmnic3
Active
Standby
Note: Team over dvUplinks with vDS
Portgroup1
VLAN 10
Portgroup2
VLAN 20
Portgroup3
VLAN 30
Portgroup4
VLAN 40
VLANs
10, 20
VLANs
30, 40
vSwitch0
Example 3: Server with 4 NIC Ports (Slight Variation)
37
Questions