Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues
-
Upload
freya-hale -
Category
Documents
-
view
84 -
download
3
description
Transcript of Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues
Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network IssuesBlaine A. Anaya XenServer Escalation Engineer05/24/2011
Overview
Agenda
XenServer Networking Architecture / vSwitch Architecture
Troubleshooting the Network
Overview
Agenda
XenServer Networking Architecture / vSwitch Architecture
Troubleshooting the Network
Performance Testing
Networking Terminology
XenServer Networking Terminology
PIF- Physical Interface Object – directly correlates to a physical interface
VIF- Virtual Interface Object- directly correlates to a virtual interface in a VM
Bridge- Represents a network and is where PIFs and VIFs are plugged in
Dom0- Short form of Domain 0 the control domain in XenServer that manages network and storage connections for virtual machines
Bond- is the association of two network interface cards to make them appear as one
Trunk – a switch port designated to carry traffic for more than one VLAN
XenServer Networking
DomUDomUDomUDomU
Xen HypervisorXen Hypervisor
Dom0Dom0
ToolstackToolstack AppApp AppApp AppAppAppApp
Native Driver /
PIF
Native Driver /
PIF
netfront/ VIF
netfront/ VIF
netfront/VIF
netfront/VIF
Guest OSGuest OS Guest OSGuest OS
BridgeBridge
Host Machine (Hardware)Host Machine (Hardware)Host Machine (Hardware)Host Machine (Hardware)
netback/0 netback/0
netback/1 netback/1 netback/2 netback/2
netback/3 netback/3
XenServer Networking Configurations- Linux Stack
Linux NIC Drivers
Linux NIC Drivers
Linux Config Files
Linux Config Files
XenServer PoolDB
XenServer PoolDB
Network Card
XAPIXAPI
Command Line
XenCenter
xsconsole
XenServer Network Terminology
Internal Switches
PIF (eth0)
VIF
VIF
VIF
Virtual Machine
Virtual Machine
Network 0 (xenbr0)
Private(xapi1)
Network Card
XenServer Network Terminology
Internal Switches
PIF (eth1)
PIF (eth0)
VIF
VIF
VIF
Virtual Machine
Virtual Machine
Network 1 (xenbr1)
Network 0 (xenbr0)
Network Card
Network Card
XenServer Network Terminology
PIF (bond0)
PIF
VIF
VIF
Virtual Machine
Virtual Machine
Network Card
Network Card
VIF
Bond 0+1 (xapi2)
PIF (eth0)
PIF (eth1)
Bonding Type (Balance SLB)
Virtual Machine
Network Card
Network Card
Virtual Machine
Bond
0:00 SEC0:10 SEC0:20 SEC0:30 SEC
Stacked Switches
Virtual Machine
Distributed vSwitch
Open Virtual Switch for XenServer
VM
HypervisorHypervisor
VM VMVMVM
HypervisorHypervisor
VM VMVMVM
HypervisorHypervisor
Visibility· Resource control · Isolation · Security
VMVM
• Open Source Virtual Switch maintained at www.openvswitch.org• Rich layer 2 feature set (in contrast to others on the market)• Ships with XenServer 5.6 FP1 as a post-install configuration option
Distributed Virtual Switch Controller
HypervisorHypervisor
HypervisorHypervisorHypervisorHypervisorHypervisorHypervisor
VMVM VM VM VM VM VM VM VM VM VM
DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches
Distributed Virtual Switch
HypervisorHypervisorHypervisorHypervisorHypervisorHypervisor
Built-in policy-based ACLs move with VMs
DVS
VMVM VM VM VM VM VM VM VM VMVM
Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123
Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123
Enabling the vSwitch
Distributed Virtual Switch
[root@vswitch1-baa-r222 ~]# xe-switch-network-backend openvswitchCleaning up old ifcfg files Remove... ifcfg-bond0 Remove... ifcfg-bond1 Remove... ifcfg-eth0 Remove... ifcfg-eth1 Remove... ifcfg-eth2 Remove... ifcfg-eth3 Remove... ifcfg-eth4 Remove... ifcfg-eth5 Remove... ifcfg-xapi2 Remove... ifcfg-xapi4 Remove... ifcfg-xenbr0 Remove... ifcfg-xenbr3Enabling openvswitch daemonConfigure system for openvswitch networkingYou *MUST* now reboot your system
#xe-switch-network-backend openvswitch (Command must be ran on each individual host)
vSwitch Architecture – Process Level View
Distributed Virtual Switch DVS Controller
OVS
Flow Table
Flow Table Cache
vSwitchNetwork A
Flow Table
Flow Table Cache
vSwitchNetwork B
ovsdb-server vswitchd
OpenFlowJSON-RPC
PIF PIF
VIF
VIF
VIF
VIF
XenServer Networking Configurations- vSwitch
Linux NIC Drivers
Linux NIC Drivers
vSwitch Config
vSwitch Config
XenServer PoolDB
XenServer PoolDB
Network Card
XAPIXAPI
Command Line
XenCenter
xsconsole
DVSCWeb Interface
Overview
Agenda
XenServer Networking Architecture / vSwitch Architecture
Troubleshooting the Network
Troubleshooting The Network
Symptoms Issue
• Intermittent Packet Loss/ Dropped Connections • Physical Connection/Switch Configuration, Bonding
• Physical Connection/Switch Configuration, Change in Hardware, Configuration Conflict.
• Network Appears Disconnected
• Bond Fails To Pass Traffic When One Leg is Disconnected
• Physical Connection/Switch Configuration, Bond Mode
• Using Command Line Interface (CLI)
• Off-line using a system status report• BareGrep Pro• Xenoscope
Troubleshooting The Network
Troubleshooting The Network
1.Check switch port configuration – Physical – Layers1-3 (Cables,NICs,Switch/Router connections)
2.Verify enabled network backend (Linux Bridge/vSwitch)
4.Use “brctl show” to see bridge/bond association.
3.Use ifconfig –a to see bonds, physical interface statistics, bridges.
5.Verify bonding configuration
6.Use ethtool for NIC settings, driver and firmware versions.
7.Use xe network-list, xe pif-list, to check XAPI configuration.
Troubleshooting the Network
Common Configuration Items to Check
Troubleshooting The Network
/etc/xensource/network.conf
/etc/sysconfig/network-scripts
/proc/net/bonding/bond0
/etc/sysconfig/iptables
Linux Bridge/vSwitch Enabled
Troubleshooting The Network
# brctl show# Shows the bridges and the interfaces plugged into them
[root@vswitch1-baa-r222 ~]# brctl showbridge name bridge id STP enabled interfacesxapi2 0000.001517868b8f no bond1 eth5 eth4xapi4 0000.001d09699d86 no bond0 eth1 eth2 vif5.0 vif6.0xenbr0 0000.001d09699d84 no eth0xenbr3 0000.001517868b8c no eth3
Linux Bridge Enabled
Troubleshooting The Network
#brctl showmacs <brname>#Shows a list of learned MAC addresses for this bridge.
[root@localhost ~]# brctl showmacs xenbr0port no mac addr is local? ageing timer 1 00:00:0c:07:ac:3c no 1.83 1 00:0c:29:3a:12:79 no 120.59 1 00:0c:29:fa:8e:e8 no 26.52
Linux Bridge/vSwitch Enabled
Troubleshooting The Network
# netstat -np# Provides information on connections and processes.
[root@vswitch1-baa-r222 ~]# netstat -npActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 127.0.0.1:37259 127.0.0.1:443 ESTABLISHED 2645/stunneltcp 0 0 127.0.0.1:36806 127.0.0.1:80 ESTABLISHED 6280/stunneltcp 0 52 10.12.45.209:22 10.54.75.163:63296 ESTABLISHED 31145/5tcp 0 0 127.0.0.1:443 127.0.0.1:37259 ESTABLISHED 6280/stunneltcp 0 0 10.12.45.209:443 10.12.45.114:39105 ESTABLISHED 6280/stunneltcp 0 0 10.12.45.209:34969 10.12.45.194:6633 ESTABLISHED 5304/ovs-vswitchd
Linux Bridge/vSwitch Enabled
Troubleshooting The Network
# netstat -s# Provides summary statistics for each protocol.
[root@vswitch1-baa-r222 ~]# netstat -sIp: 17340461 total packets received 9190 with invalid addresses 0 forwarded 0 incoming packets discarded 12463755 incoming packets delivered 14230986 requests sent out 8 dropped because of missing route
Tcp: 69504 active connections openings 126760 passive connection openings 0 failed connection attempts 229 connection resets received 17 connections established 12462000 segments received 13220998 segments send out 3144 segments retransmited 0 bad segments received. 416 resets sent
Linux Bridge/vSwitch Enabled
Troubleshooting The Network
#ethtool –k <interface>#Provides information on current offload settings
[root@vswitch1-baa-r222 ~]# ethtool -k eth0Offload parameters for eth0:rx-checksumming: ontx-checksumming: onscatter-gather: ontcp-segmentation-offload: onudp-fragmentation-offload: offgeneric-segmentation-offload: ongeneric-receive-offload: offlarge-receive-offload: off
Linux Bridge/vSwitch Enabled
Troubleshooting The Network
#ethtool –i <interface>#Provides information on driver/firmware versions for network cards
[root@vswitch1-baa-r222 ~]# ethtool -i eth0driver: bnx2version: 2.0.8efirmware-version: bc 2.9.1bus-info: 0000:04:00.0
vSwitch Enabled
Troubleshooting The Network
#ovs-appctl bond/list#Shows Bridge, Bond, Slave Association
[root@vswitch1-baa-r222 ~]# ovs-appctl bond/listbridge bond slavesXapi2 bond1 eth4, eth5Xapi4 bond0 eth2, eth1
Disclaimer: Using OVS command line options for configuration purposes is not supported. The vSwitch should only be configured using XenCenter, xe CLI, xsconsole, and the Distributed vSwitch Controller.The commands shared here are for data collection and diagnostic purposes only.
vSwitch Enabled
Troubleshooting The Network
#ovs-appctl bond/show bond0 #Shows bond members, up/down delay, and next rebalance time.
[root@vswitch1-baa-r222 ~]# ovs-appctl bond/show bond0updelay: 31000 msdowndelay: 200 msnext rebalance: 4314 msslave eth2: enabled
active slavehash 123: 1 kB load
86:43:b2:1a:f2:d0slave eth1: enabled
vSwitch Enabled
Troubleshooting The Network
#ovs-appctl fdb/show <bridge_name> #Shows MAC Table/VLAN information for the bridge
[root@vswitch1-baa-r222 ~]# ovs-appctl fdb/show xapi4 port VLAN MAC Age 3 0 00:1d:09:2c:c4:c9 58 3 0 0a:34:ee:08:53:06 47 3 0 6a:e8:14:89:5c:af 42 3 0 ba:89:bf:f5:b8:ab 35 3 0 00:16:c8:d8:f1:11 27
vSwitch Enabled
Troubleshooting The Network
#ovs-ofctl dump-flows <bridge_name> #Shows FlowTable – (ACLs applied from controller)[root@vswitch1-baa-r222 ~]# ovs-ofctl dump-flows xapi4 | grep dropMay 02 15:49:07|00001|ofctl|INFO|connecting to unix:/var/run/openvswitch/xapi4.mgmtcookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=32763, n_packets=0, n_bytes=0, tcp,dl_dst=86:43:b2:1a:f2:d0,nw_dst=10.12.45.151,tp_src=80,actions=dropcookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=65529, n_packets=15, n_bytes=930, tcp,in_port=4,dl_src=86:43:b2:1a:f2:d0,nw_src=10.12.45.78,tp_dst=80,actions=drop
vSwitch Enabled
Troubleshooting The Network
#ovs-dpctl dump-flows <bridge_name> #Shows FlowCache – (ACLs applied from controller)[root@vswitch1-baa-r222 ~]# ovs-dpctl dump-flows xapi4 | grep mac86:43
tunnel00000000:in_port0004:vlan65535:pcp0 mac86:43:b2:1a:f2:d0->00:00:0c:07:ac:3c type0800 proto6 tos0 ip10.12.45.78->69.147.112.160 port4284->80, packets:1, bytes:62, used:2.160s, actions:drop
vSwitch Enabled
Troubleshooting The Network
#ovs-appctl vlog/list#Show current logging levels[root@vswitch1-baa-r222 ~]# ovs-appctl vlog/list console syslog file ------- ------ ------bridge EMER ERR INFOvswitchd EMER ERR INFOxenserver EMER ERR INFOofproto EMER ERR INFOsflow EMER ERR INFOjsonrpc EMER ERR INFOfail_open EMER ERR INFOnetflow EMER ERR INFOovsdb_error EMER ERR INFO
vSwitch Enabled
Troubleshooting The Network
#vlog/set module[:facility[:level]] #Modify vswitch logging level
Sets the logging level for module in facility to level:
•Module may be any valid module name (as displayed by the --list action on ovs-appctl(8)), or the special name ANY to set the logging levels for all modules.
• Facility may be syslog, console, or file to set the levels for logging to the system log, the console, or a file respectively, or ANY to set the logging levels for both facilities. If it is omitted, facility defaults to ANY.
Note: The log level for the file facility has no effect unless ovs-vswitchd was invoked with the --log-file option.
•Level must be one of emer, err, warn, info, or dbg, designating the minimum severity of a message for it to be logged. If it is omitted, level defaults to dbg.
Status Report and BareGrepPro
Troubleshooting The Network – Off-Line
Status Report and Xenoscope
Troubleshooting The Network – Off-Line
Status Report and Xenoscope
Troubleshooting The Network – Off-Line
Status Report and Xenoscope
Troubleshooting The Network – Off-Line
• CTX127885 - Introduction to XenServer Networking
• CTX123489 - XenServer VLAN Networking
• CTX124421 - Understanding Network Interface Card Bonds in XenServer
• CTX127970 - Distributing Guest Traffic Over Physical CPUs in XenServer
• CTX127065- XenServer Virtual Machine Performance Utility
• CTX123477 - How to Move a XenServer Pool to a Different IP Subnet
• CTX125358 - How to Identify the Network Adapters on XenServer
• CTX101810 - Communication Ports Used By Citrix Technologies
Useful Networking CTX Articles
Q & A
Before you leave…
• Recommended related breakout sessions: • Session: YN203: Managing VM networking across the datacenter with XenServer distributed
virtual switching Date: Wednesday May 25th Time: 4:30-5:15 Room: Moscone 2003-2005
• Session surveys are available online at www.citrixsummit.com starting Thursday, May 26• Provide your feedback and pick up a complimentary gift at the registration desk
• Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account