Deploying and Troubleshooting the Nexus 1000v...
Transcript of Deploying and Troubleshooting the Nexus 1000v...
Deploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere
BRKVIR-3013
Matthew Wronkowski – Technical Leader Virtualization Services
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Agenda
• Current N1K Releases and New Features
• Licensing
• Virtual Supervisor Module (VSM) & VEM
• VSM High Availability
• Upgrades
• Port-Profiles & Port Channels
• VXLAN
• Cisco Cloud Services Platform / Nexus1x10
3
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Nexus 1000V Virtual Switch | Build & Price
4
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Virtual Networking and Cloud Network Services
Nexus 1000V
• Distributed Switch
• NX-OS consistency
VSG
• VM-level controls
• Zone-based FW
ASA 1000V
• Edge firewall, VPN
• Protocol Inspection
vWAAS
• WAN optimization
• Application traffic
WAN
Router
Servers
Tenant A ASA
1000V
Cloud
Firewall
Nexus 1000V Physical
Infrastructure
Virtualized/Cloud Data Center
vWAAS
Cisco
Virtual
Security
Gateway
Switches
Cloud Network Services
Citrix
NetScaler
VPX
Imperva
SecureSphere
WAF Cloud
Services
Router
1000V
Zone A
Zone B
vPath VXLAN
Multi-Hypervisor (VMware, Microsoft, Ubuntu, RedHat*)
Network
Analysis
Module
(vNAM)
vNAM
• App Visibility (L2-L7)
CSR 1k
• WAN GW
• Routing & VPN
Ecosystem
• Citrix NetScaler
• Imperva Web FW
5
“Name a feature we will not implement on Nexus 1000V.”
Saravan Rajendran, Cisco CNSG VP
6
Current Releases and New Features
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Current Nexus 1000V Releases
• ESX – 5.2(1)SV3(1.1)*
– 256 VEMs, 12K vEth count
– VXLAN 2.0 (BGP)
– N1K Management Center
• ESX – 4.2(1)SV2(2.2)
– Dynamic Fabric Automation Leaf
– VDP – VSI Discovery Protocol
– Universal Licensing
• ESX - 4.2(1)SV2(2.1a)
– Scalability Release – 128 VEMs
– VXLAN 1.5, VXLAN GW
– Geographically Separated VSMs
– Removed ESX 4.1 support
• Hyper-V – 5.2(1)SM1(5.2a)
– SCVMM 2012 SP1 & R2
– Windows Server 2012 & R2
– VSG VM and Custom Attributes
– Universal Licensing
• InterCloud – 5.2(1)IC1(1.2)
– Simplified Platform Image
– Local License Server or Cisco PNSC
• Ubuntu KVM / OpenStack
– Initial Release
8
*Next Release
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Evolution of VXLAN to version 1.5
• Unicast mode
– Simplifies VXLAN deployment
– Reduces network dependency (no multicast)
– Easier troubleshooting
– Flood directly to VXLAN Tunnel End Points (VTEP)
• Unicast Mac-address Distribution Mode
– Flooding is eliminated
– VSM learns all MACs and programs mappings to VEMs
– Faster response time
– Will not support VXLAN veth trunking(multi-mac)
– Requires static MACs (won’t work with MS NLB)
9
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
vTracker Feature
• Provides intuitive virtualization perspective to the network-admin
• Pulls data from vCenter and VEM
– Gives “cloud” view of connected objects
• Enabled with “feature vtracker”
• There are 5 view options
– module-view
– upstream-view
– vlan-view
– vm-view
– vmotion-view
10
SV2# show vtracker vm-view info vm win3
Module 5:
VM Name: win3
Guest Os: Microsoft Windows Server 2003
Standard (32-bit)
Power State: Powered On
VM Uuid: 423ca4df-26d0-50c1-d531-1a49b3a83aed
Virtual CPU Allocated: 1
CPU Usage: 0 %
Memory Allocated: 1024 MB
Memory Usage: 7 %
VM FT State: Unknown
Tools Running status: Running
Tools Version status: current
Data Store: datastore1 (2)
VM Uptime: 25 days 3 hours 56 minutes 15s
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Nexus 1000V Manager – Installation Screenshot
• Zero CLI – full GUI interface
• Auto Host Selection
• Deploy Redundant VSMs
• Best Practices Auto-Implemented
• Automated prompts with suggestion for alternatives
• Customize Installation for Advanced Users
• *Available Summer 2014
Install / Migrate / Upgrade / Monitor
11
Licensing Info
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Licensing – Essential Edition (No Expiration)
• Default mode for New Installs
• All features except…
– Cisco TrustSec (CTS)
– DHCP Snooping
– IP Source Guard / Dynamic ARP Inspection
– Virtual Security Gateway (VSG)
– VXLAN Gateway
• 128 modules with 4096 virtual ports
• Support Options
– Pay Nothing – support is through the communities site off cisco.com • https://communities.cisco.com/community/technology/datacenter/nexus1000v
– Pay for service contract
13
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Licensing – Advanced Edition
• For customers that want more security features
• Customers with existing licenses will be considered Advanced
• Upgrade process will migrate VSM to Advanced Edition
• Required for VXLAN Gateway and VSG
• Licensed customers can get Virtual Security Gateway(VSG) for free
– Cisco Account Team can submit request
– VSG will no longer be sold separately
• 256 modules with 12k virtual ports (SV3)*
• 60-day Trial after which Advanced FeatureSet is disabled
14
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Universal Licensing
• A common license is shared for both N1k & VSG.
• Cross Hypervisor portability.
• The license name is NEXUS1000V_LAN_SERVICES_PKG.
• Following upgrade, request a new Permanent license within 60 days.
15
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Licensing – New Commands
• Display Current Edition switch# show switch edition
• To switch between Essential or Advanced switch(config)# svs switch edition [essential | advanced]
• VEM Licenses are Sticky – Removed & Offline VEMs hold a license switch# show module vem license-info
Licenses are Sticky
Mod Socket Count License Usage License Version License Status
--- ------------ ------------- --------------- --------------
3 2 2 1.0 licensed
• VEM license transfer to pool: switch(config)# svs license transfer src-vem <module> license_pool
16
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Licensing – Overdraft Licenses
• Extra licenses to use in temporary situations
• 16 extra sockets
– Sometimes more depending on number of licenses you’ve purchased
• Can only be used after a valid license is installed
• No penalty
– Full TAC Support for Overdraft Modules
17
SV2# show license usage NEXUS1000V_LAN_SERVICES_PKG
----------------------------------------
Feature Usage Info
----------------------------------------
Installed Licenses : 16
Default Eval Licenses : 0
Max Overdraft Licenses : 16 <----
Installed Licenses in Use : 12
Overdraft Licenses in Use : 0 <----
Default Eval Lic in Use : 0
Default Eval days left : 0
Licenses Available : 20 <---- 4 + 16
Shortest Expiry : 04 Feb 2015
Virtual Supervisor Module Deployment and Troubleshooting
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Nexus 1000V Architecture
Hypervisor Hypervisor Hypervisor
VEM-N VEM-1 VEM-2
VSM: Virtual Supervisor Module
VEM: Virtual Ethernet Module Server
Admin
NX-OS
Data Plane
VSM-1 (active)
VSM-2 (standby)
Virtual Appliance
NX-OS
Control Plane Network
Admin
Modular Switch
… Linecard-N
Supervisor-1 (Active)
Supervisor-2 (StandBy)
Linecard-1
Linecard-2
Ba
ck P
lan
e
19
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Virtual Supervisor Module (VSM)
• VSM is a Virtual Machine
– On ESXi, Hyper-V, Ubuntu KVM / OpenStack
– On Nexus 1x10 / Cloud Services Platform
• Control plane for the Nexus 1000V solution
– VEM packet forwarding not impacted by reloads
• Responsible for
– Programming and Managing Virtual Ethernet Modules (VEM)
– Communicating with Management Applications • VMware vCenter, SCVMM, Horizon Dashboard
• 1 VSM HA pair can manage 128 VEMs
• Coexist with VMware vSwitch, vDS, Microsoft Logical, Native Switches
20
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Nexus 1000V VSM Interfaces
• Control
– L2/L3 VEM (AIPC)
– VSM-VEM Heartbeats (L2/L3)
– VSM-VSM Synchronization (L2)
– VSM-VSM HA Heartbeats (L2/L3)
• Packet
– CDP, IGMP, NetFlow, SNMP
• L3 Mode
– Collapsed Ctrl, Pkt into mgmt0
– VSM-VEM flow from mgmt0
– Dedicated Control: svs mode L3 interface [control | mgmt0]
• Management
– SSH console access
– SNMP, HTTP
– vCenter Communication
– HA Heartbeat Backup
• Interface Order is always the same!
VSM-P eth0: control
eth1: mgmt0
eth2: packet
21
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Deployment Scenarios
• Supports the VSM on a VEM
• Supports the VSM on any hypervisor native, logical, or distributed switch
• Supports the VSM on any supported hypervisor (ESXi/Hyper-V/N1110)
• Keep VSMs on different physical hosts
– Use anti-affinity rules
• Storage wise we don’t care.
– VSM can be hosted on network storage
22
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Stretched Nexus 1000V Model
• VSMs and VEMs spread across Datacenters
• VSMs can be split across DCs
– Requires L2 connectivity across DCI
– 10ms latency across DCI
• Not supported with Hyper-V
– Supported in a future release
23
VSM
VSM
hypervisor
VEM-1
VM VM VM
Local DC
hypervisor
VEM-2
hypervisor
VEM-4
VM VM VM
hypervisor
VEM-3
VM VM VM
Remote DC
VM VM
DCI
VM
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Control Modes
• L3 Mode
– L3 is the recommended & default • Easier to troubleshoot
• Flexible
– Requires an IP address be assigned to the VEM
– Uses UDP4785 for both source and destination
– Sourced from mgmt0 by default
• L2 mode
– Requires L2 connectivity through control0 interface to all VEM modules
– L2 still supported on ESX
– Not supported with Hyper-V or KVM
24
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM L3 Configuration and Planning
• Two options for the L3 control interface
– mgmt0 (default)
– control0
• Use Control0 to separate control and management traffic
• Mgmt and Control use different VRF
– mgmt0 uses VRF management
– control0 uses VRF default
• Primary and Secondary VSM still need to be L2 adjacent!
– Test with mping broadcast command. 0x201 is control between VSMs
25
# mping broadcast
64 bytes from node 0x0201 (msg id = 0x030b1e 1) (time=0 sec, 1510 usec)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Connectivity to VMware vCenter
• VSM connects to vCenter using SSL connection
– VC Extension contains the SSL cert
– Unique extension ID for the VSM
– Ability to generate own certificates
• VSM talks to vCenter using its API
– We push and pull data to/from vCenter
• VSMs get tied to a VMware Datacenter
– Multiple VSMs tied to same DC is allowed
– VSM can manage across clusters but not datacenters • Can get confusing
26
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Connectivity Errors - ESXi
• If you get “Extension key was not registered before it’s use”
– Re-register the Extension Key with VMware vCenter
• If you get “Connection refused. connect failed in tcp_connect()”
– Ping vCenter IP from VSM CLI
– VMware admin could have changed the http port
– API communication is through port 80 with VMware vCenter
– Find new port and change it on VSM
27
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM and vMotion/Live Migration
• Manual vMotion/Live Migration is supported
• VMware DRS is NOT recommended for Primary & Secondary VSMs
• Aggressive settings could lead to excessive VSM-VEM heartbeat packet drops
• Best practice to keep Primary and Secondary VSM outside DRS control
• Use anti-affinity rules where possible
28
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Backing up the VSM
• A running-config is not enough to restore
• VSM on ESXi
– Clone to a template
– You can restore from a template and saved-config
– Must be powered down
• VSM on Nexus 1x10
– Export a VSM to a file
– Import the saved VSM to restore
• VSM on ESXi Snapshots
– Not officially supported
– I/O latency cost associated with expanding the differential file
29
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Best Practices - Summary
• L3 control is the preferred method
• Use mgmt0 for control traffic
• Primary and Standby VSM in same L2 domain!!!
– Required even if VSMs are split between datacenters
• VSM on VEM is supported
• 10ms Latency between components: VSM-VSM, VSM-VEM
– 10ms even for VSMs split between datacenters
– For VEMs at branch locations 100ms
• Backup your config!!!
30
Nexus 1000V High Availability
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Redundancy Manager
• HA had to evolve to support split datacenter VSMs
• New Redundancy Manager process polls:
– VEM Manager – polls for number of active VEMs attached to VSM
– VMS process – retrieves which VSM has active VC connectivity
– SNMP Library – gets the last configuration time
• Runs on both primary and secondary VSM
• Heartbeats
– VSM-VSM every second. Drop after 6 missed
– VSM-VEM every second. Drop after 15 missed
32
SV2# show system internal redundancy trace
1 0s START_THREAD ST_NP ST_NP ST_INVALID
2 0s CP_STATUS_CHG ST_INIT ST_NP ST_INIT
3 0s SET_VER_RCVD ST_INIT ST_NP ST_INIT
4 0s STATE_TRANS ST_INIT ST_INIT ST_INIT EV_OS_INIT ST_AC_INIT
5 0s CP_STATUS_CHG ST_AC ST_INIT ST_AC_INIT
6 0s STATE_TRANS ST_AC ST_SB ST_AC_INIT EV_OS_SB ST_AC_SB
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Split Brain Recovery for ESXi
• Redundancy Manager in SV2(2.2)
– Module Count
– vCenter Status
– Last Configuration Time
– Last Standby-Active Switch(VSM with longer “primary” active time)
– Out-of-Sync / Split-Brain causes VSM to reload
33
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
When does a VEM switch VSMs?
• What if we have two active VSMs?
• What causes a VEM to switch?
– Standby VSM becomes active and broadcasts to all VEMs
– VEM will attach depending on • Connectivity between VEM and VSM
• VEM receives the “request to switch”
• VEM goes into headless mode after 15 seconds
• If a VEM is headless traffic forwarding continues!
– vMotion/Live Migration is blocked
34
Upgrades
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Upgrades
• First always read and follow the upgrade guides
– Go in order
• Take a backup of the VSMs
– On ESXi use the clone to template option
– On Nexus 1x10s use the export function
– Backup the running-config
• Generate a Tech-Support before the upgrade
• If something goes wrong STOP and call TAC
• Use a maintenance window
– VEM upgrades require ESXi hosts to be in Maintenance Mode
36
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Supported Upgrades
Starting Version 1.4 1.5 2.1 2.2 Combined
VMware Upgrade
Notes
1.3 Yes 1.4 first* 1.4 first* 1.4 first* No
1.4 Yes Yes Yes No 1.4 last version
supporting
ESX 4.0
1.5 Yes Yes Yes 1.5.2 for
combined
2.1 Yes Yes 2.1 last version
supporting
ESX 4.1
37
Upgrade matrix: http://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.html
* Must upgrade to 1.4b first
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Upgrades to 2.2
• Scalability limits may require changes to the VM settings
• For full scalability support:
– CPU reservation to 2GHz
– Memory to 3GB
– VSMs do NOT support multiple vCPUs
• Steps
– Shutdown Secondary VSM
– Make VM changes
– Power Secondary on
– System Switchover
– Repeat steps on Primary VSM
• API can be upgraded individually now
– “show plugin status”
38
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Upgrading the VSM
• Changes from 2.1
– VSMs can run newer software than VEMs. New features disabled until VEMs upgraded.
• ISSU upgrade is similar to other Nexus switches
– Copy new kickstart and system images to bootflash
– Run “install all” command • Verifies software compatibility
• Copies images to secondary’s bootflash.
• Upgrade/Reboot the Secondary VSM
• Switchover to Secondary VSM – It’s now the active VSM with VEMs attached
• Upgrade/Reboot the old-Primary VSM
• Requires no outage of the VSM
• Change CPU/Memory after the SV2(2.2) upgrade is complete
39
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Troubleshooting VSM Upgrades
• If something is wrong after the VSM upgrade STOP
– Call TAC
– Rollback using backup method • Shutdown the VSM VMs
• Power-on the Clones (ESXi), Import the backup (Nexus 1x10)
• Changing boot variables to older image is not supported but often works
• Sometimes the VEM won’t connect to the Standby VSM
– Try a “system switchover” once the old primary is upgraded
• Might want to verify Standby VSM before upgrade
– Make sure VEMs can connect to standby
– Use “system switchover” command
40
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Upgrading the VEMs
• VEM module upgrade kicked off on VSM
– If VUM is installed everything is automatic • VSM communicates with vCenter to manage the upgrade
• Host is placed in maintenance mode(if DRS is installed VMs are migrated off)
• VEM is upgraded and host exits maintenance mode
• Moves on to the next host
– If VUM is not installed • Still initiate the process on the VSM
• User manually places ESXi hosts in maintenance mode
• Upgrade the VEM with esxcli command
• Exit maintenance mode and move to the next host
• Always complete the upgrade
– Issue the “vmware vem upgrade complete” command
– Signals vCenter to use the new VEM VIB when hosts are added
41
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Troubleshooting VEM Upgrades
• Remember the VMware admin has to acknowledge upgrade in vCenter
• Don’t upgrade the VEMs by pushing a baseline
• Make sure you have DRS capacity
– Need to be able to handle one ESXi host in maintenance mode
• If a particular ESXi host fails
– It’s usually because the host cannot go into maintenance mode
– From vCenter attempt to put the host in maintenance mode • Troubleshoot any issues that prevent it
– If an ESXi host is running a vCenter VM this can cause problems
• You can restart the VEM upgrade after it fails
– It will only upgrade hosts that did not succeed
42
Virtual Ethernet Module Deployment and Troubleshooting
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Deployment – Best Practices
• Again we recommend L3 Control
• L3 control requires a VMKernel NIC on N1K DVS
– We need an L3 interface to forward control traffic
– 10/100ms latency for local vs. branch office
• Recommend using the ESXi management VMKernel NIC
– Requires management interface to the VEM
– Doesn’t require static routes on ESXi hosts
• Don’t create an L3 vmk on same subnet as mgmt vmk
• Don’t use UCS “Dynamic vNICs” in Service-Profiles
– VEM and VM-FEX are mutually exclusive
44
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Deployment – vEth Port-Profile
• vmk0 interface needs to be migrated to this port-profile
• It must have capability l3control and system VLAN
• Each VMKernel VLAN needs a different port-profile
• VSM only permits VMKs to connect to this port-profile
45
port-profile type vethernet vmk-l3
capability l3control
vmware port-group
switchport mode access
switchport access vlan 119
capability vxlan
no shutdown
system vlan 119
state enabled
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Deployment – Uplink Port-Profile
• Typically a trunk – Verify upstream switch allowed VLAN list matches
• Must have system vlans & a port-channel defined
• MTU must match. Especially important when using OTV.
46
port-profile type ethernet system-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 119,199,219,319
mtu 9000
channel-group auto mode on mac-pinning
no shutdown
system vlan 119,319
state enabled
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM L3 Troubleshooting
1. VMK migrated behind VEM?
2. VSM-ESXi connectivity?
• Static route needed?
3. L3 vEth Port-Profile correct?
4. Uplink Port-Profile correct?
5. Check the Opaque Data
6. Check Heartbeats
47
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Troubleshooting – VSM Connectivity
• VEM adds in vCenter but does not show up on VSM “show module”
• With L3 its usually an IP routing problem
– If you can ping from VSM to VMK interface then VEM should connect.
– Troubleshoot as you would all VMware L3 issues
• With L2 most of the time its a Control VLAN issue
– Verify Control VLAN connectivity in upstream network
– Check upstream switches for VEM AIPC MAC address
• Additional Information in Appendix 2
48
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Deployment – VMKs on same subnet
• Don’t use multiple VMKs on the same subnet on different virtual switches
• VMware uses a single TCP/IP stack for all VMK interfaces
• No way to pin traffic to an uplink interface.
• One interface gets picked for all traffic on that subnet
– Check out VMware KB article 2010877
• Only one gateway per host
50
VMware ESX
VEM-1
VMK1
192.168.10.200
VMK0
192.168.10.100
vSwitch
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Setting Verification
• Verify the VRF
• Can the VSM ping the VEM
• Check SVS domain
52
SV2# show ip route vrf management
0.0.0.0/0, ubest/mbest: 1/0
*via 14.17.119.254, mgmt0,
[1/0], 6d20h, static
SV2# ping 14.17.219.22
PING 14.17.219.22 (14.17.219.22): 56 data bytes
64 bytes from 14.17.219.22: icmp_seq=0 ttl=62 time=1.254 ms
64 bytes from 14.17.219.22: icmp_seq=1 ttl=62 time=1.057 ms
64 bytes from 14.17.219.22: icmp_seq=2 ttl=62 time=1.055 ms
SV2# sh svs domain
SVS domain config:
Domain id: 1919
Control vlan: NA
Packet vlan: NA
L2/L3 Control mode: L3
L3 control interface: mgmt0
Status: Config push to
VC successful.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Check Opaque Data
• Opaque data is bootstrap information for the VEM
– Pushed via SCVMM or vCenter during “Host Add to DVS”
• Is the right Opaque data getting pushed to the ESXi host?
53
Should match VLAN defined
in vEth Port-Profile
Should match MAC of
control 0 or mgmt 0
~ # vemcmd show card Card UUID type 2: 9aed7c30-84f8-11e2-1234-ff987600005f
Card name:
Switch name: SV2
Switch alias: DvsPortset-0
Switch uuid: b2 40 3c 50 72 8e 15 f5-6a 3c 7f d1 c6 13 70 cd
Card domain: 1919
Card slot: 3
VEM Tunnel Mode: L3 Mode
L3 Ctrl Index: 49
L3 Ctrl VLAN: 119
VEM Control (AIPC) MAC: 00:02:3d:17:7f:02
VEM Packet (Inband) MAC: 00:02:3d:27:7f:02
VEM Control Agent (DPA) MAC: 00:02:3d:47:7f:02
VEM SPAN MAC: 00:02:3d:37:7f:02
Primary VSM MAC : 00:02:3d:70:1f:07
Primary VSM PKT MAC : 00:02:3d:70:1f:08
Primary VSM MGMT MAC : 00:02:3d:70:1f:06
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
View Heartbeat Messages on VEM
• Use vempkt on the ESXi host vempkt capture [egress|ingress] vlan 119 ltl 50
– Run for 10s to capture several heartbeat cycles
vempkt cancel capture all
vempkt display detail all
• vempkt can now export to a pcap file vempkt pcap export <filename>
• Look for heartbeat messages on VSM
54
SV2# show module vem counters
--------------------------------------------------------------------------------
Mod InNR OutMI InMI OutHBeats InHBeats InsCnt RemCnt Crit Tx Errs
--------------------------------------------------------------------------------
3 5086 2 2 593401 348535 2 1 0
4 5 4 4 593401 593296 4 3 0
5 0 0 0 593401 0 0 0 0
6 105 4 4 593401 591303 4 3 0
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VEM Troubleshooting - vemlog
• Used for detailed debugging of programming and packet flows
• Executed on the Hypervisor Host
• Enable different debug options to help troubleshoot
– LACP
– QOS
– VXLAN
– IGMP
– VSM<-->VEM Data
• http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml
55
~ # vemlog show debug | grep lacp
Module Available Printing
sflacp ENWID PL (223) ( 0)
sf_lacp_pdu_utils ENWID PL (223) ( 0)
sflacp_hostdata ENWID PL (223) ( 0)
~ # vemlog debug sflacp all
~ # vemlog show debug | grep lacp
sflacp ENWID PL (223) ENWIDTPL (255)
sf_lacp_pdu_utils ENWID PL (223) ( 0)
sflacp_hostdata ENWID PL (223) ( 0)
Port-Profiles Deploying and Troubleshooting
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port-Profiles
Usage
<type>
Port-Profile Port-profiles
vEthernet
VM vmk l3control
/ vservice
Ethernet
UPLINK
vEthernet PP (default)
-Virtual Interfaces (vEth x/) (VMs, VMK)
-Typically Access Ports
-Configuration: VLAN, ACL, Pinning, QoS
Ethernet PP
-Physical Interfaces (Eth x/y)
-Typically Trunk (could also be access)
-Configuration: Port-Channel, ACLs, QoS
57
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Switch Interface Types
• Ethernet Port (eth)
– Correspond to the physical NIC interfaces leaving the server
– Specific to each “module” or VEM
– VMware’s vmnicX == Cisco ethx/y
– Up to 32 physical ports supported per host
• Port Channel (port-channel)
– Aggregation of physical Ethernet ports
– Up to eight Port Channels per host
• Virtual Ethernet Port (vEth)
– One per virtual NIC interface (vNIC) including service console / vmknic
– Notation is VethX
– No module number is assigned to keep naming persistent as VMs move between modules (hosts/VEMs)
58
VM1 VM2
Eth3/1 Eth3/2
Po1
Veth2 Veth1
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Loop Prevention without STP
59
Cisco VEM
VM1 VM2 VM3 VM4
Cisco VEM
VM5 VM6 VM7 VM7
Cisco VEM
VM9 VM10 VM11 VM12
BPDUs are Dropped
Eth4/1 Eth4/2
X
No Switching from
Physical NIC
to NIC
déjà vu check
Frames with local
MAC Dropped on
Ingress
X
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Spanning-tree and BPDU – Best Practice
• Mandatory Spanning-Tree settings per port
– IOS set STP portfast • cat65k-1(config-if)# spanning-tree portfast trunk
– NXOS set port type edge • n5k-1(config-if)# spanning-tree port type edge trunk
• Highly Recommended Global BPDUFilter/BPDUGuard
– IOS • cat65k(config)# spanning-tree portfast bpdufilter
• cat65k(config)# spanning-tree portfast bpduguard
– NXOS • n5k-1(config)# spanning-tree port type edge bpduguard default
• n5k-1(config)# spanning-tree port type edge bpdufilter default
• BPDU Filter is mandatory for LACP port-channels
• Set per-port BPDU Guard when Global is not possible
60
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Ethernet (uplink) Port-Profile Troubleshooting
• Port-Profiles with multiple NICs need a port-channel
– Causes duplicate packets
– Kicks in déjà vu driver • Requires extra CPU processing
• Fills the logs
– When in doubt, use mac-pinning
• Also same issue if you overlap VLANs in different Port-Profiles on same host
61
port-profile type ethernet uplink-nopc
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-3967,4048-4093
no shutdown
system vlan 11
state enabled
port-profile type ethernet uplink-nopc
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-3967,4048-4093
channel-group auto mode on mac-pinning
no shutdown
system vlan 11
state enabled
WRONG RIGHT
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Nexus 1000V System VLANs
• System VLANs enable interface connectivity before an interface is programmed
• System port-profiles become part of the opaque data
– VEM will load system port-profiles and pass traffic even if VSM is not up
– Unprotected (No ACLs, VSG) before module registers for first time
• Addresses chicken and egg issue
– VEM needs to be programmed, but it needs a working network for this to happen
• Port profiles that contain system VLANs are “system port profiles”
– Allowed 32 port-profiles with system VLAN
62
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
System VLAN Guidelines
• The system VLAN must be a subset of the allowed VLAN list on trunk ports
• Only one system VLAN on an access port
• The ‘no system vlan’ command only when no interface is using the profile
• Once a system profile is in use by at least one interface
– Can add to the list of system VLANs
– Cannot delete VLANs from the list – reason to limit usage
• System vlans must be set on egress and ingress port-profiles
• Required System VLANs
– Control, Packet, IP Storage, VMKernel, vCenter, any Management Networks
63
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VMware DVS Max-Port Issues
• Default to 32 max-ports per port-profile
• Counts toward the maximum number of VMware DVS ports
– 8192 by default
– Pre-Provisioned
– Some ports are consumed when you add an ESX host to the DVS
• Two methods to remedy:
– Max-ports under “svs connection <name>” • Allows you to increase the ports of the VMware DVS
– Port-binding “auto expand” in veth port-profiles • N1KV dynamically adds ports as VMs are added
• Set port-binding as default with “port-profile default port-binding static auto expand”
64
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Microsoft Network Load Balancing Support
• Unicast mode is officially supported method
– “no mac auto-static-learn” in vEth port-profile
• Multicast Mode
– NLB virtual cluster address requires a static ARP entry on the edge router
– Works through flooding
• Multicast Mode IGMP
– Disable IGMP snooping on the N1KV
– Upstream switches enable IGMP snooping
– Enable IGMP Querier in the environment
– NLB virtual cluster address requires a static ARP entry on the edge router
– CSCue32210 - Add support for Microsoft NLB - Multicast+IGMP
65
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Jumbo Frames Support
• System jumbo mtu 9000 – Enabled globally by default in SV1(4)+
– Sets the systemwide jumbo MTU size
– Generally do not need to change
• vEthernet ports are 9000 by default
• MTU setting for “ethernet” type port-profile
– Simply use “mtu size” in port-profile and nothing else
• Still need to configure upstream network devices
– UCS System QoS Class
– UCS vNIC QoS Policy
– Nexus 5k / 7k / etc
66
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port-Profile Using Weighted QOS
• Configuration Steps to limit vMotion traffic
68
n1kv-l3(config)# class-map type queuing match-all vmotion-class
n1kv-l3(config-cmap-que)# match protocol ?
n1k_control N1K control traffic
n1k_mgmt N1K management traffic
n1k_packet N1K inband traffic
vmw_ft VMware fault tolerance traffic
vmw_iscsi VMware iSCSI traffic
vmw_mgmt VMware management traffic
vmw_nfs VMware NFS traffic
vmw_vmotion VMware vmotion traffic
n1kv-l3(config-cmap-que)# match protocol vmw_vmotion
n1kv-l3(config-cmap-que)# policy-map type queuing vmotion-policy
n1kv-l3(config-pmap-que)# class type queuing vmotion-class
n1kv-l3(config-pmap-c-que)# bandwidth percent 50
n1kv-l3(config)# port-profile type eth uplink-vpc
n1kv-l3(config-port-prof)# service-policy type queuing output vmotion-policy
Port Channels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels
• LACP Port-Channels
– Requires upstream switch support and configuration
• VPC – MAC Pinning
– Works with any upstream switch
– Allows for pinning of vEths (VM) to specific links
• VPC – Host Mode CDP/Manual (deprecated)
– NIC association is either Manual or CDP
70
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels
• Best Practice Configuration Guide
– http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml
• All Ethernet Port-Profiles must be configured in a Port-Channel
• LACP & MAC-Pinning are recommended modes
– Use Manual/Static Pin Group for granular traffic steering
– Use Manual/Static Pin Groups with multiple vMotion VMKs in ESX 5.x
• Same link-speed for all members. No mixing 1G+10GE+40GE interfaces.
71
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels – Best Practice
• If the upstream switch can be clustered (VPC, VBS Stack, VSS) use LACP
• If you are using LACP also use LACP Offload
• UCS-B must use MAC-Pinning
• If the upstream switch can NOT be clustered use MAC-PINNING
• Create channel-groups in port-profile
– Let VSM build the interface port-channel & add physical NICs
• All physical switch ports in port-channel configured identical
72
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels – MAC Pinning
• MAC Pinning provides the dynamism of vPC Host-Mode without requiring CDP to be configured on the upstream switch
vSphere
VM VM VM VM
sys-uplink
The VM MAC address is used to select link.
port-profile type ethernet uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-10
channel-group auto mode on mac-pinning
no shut
state enable
system vlan 10
73
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels – MAC Pinning (Link Failure)
• If a failover occurs, all the traffic pinned to an interface will be migrated to the other interfaces. VEM sends GARP to flush upstream CAM tables.
vSphere
VM VM VM VM
sys-uplink
The VM MAC address is used to select link.
port-profile type ethernet uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-10
channel-group auto mode on mac-pinning
no shut
state enable
system vlan 10
74
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels – MAC Pinning
• Use Network State Tracking (NST) to detect non-link failures
• Each Eth interface added is a unique Service Group
– SGID # assigned based off vmnic#
• Use “pinning id” command under vEthernet Port-Profile
– Pins the VM to a particular uplink
– Ordered list for backup
– n1kv(config-port-prof)# pinning id 0 backup 1 2
• Default assignment is Round Robin to an SGID
• New command to make SGID # relative
– n1kv(config-port-prof)# channel-group auto mode on mac-pinning relative
75
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
MAC Pinning (Host Pinning Tables) n1kv# sh port-channel summary
1 Po1(SU) Eth NONE Eth5/1(P) Eth5/2(P)
2 Po2(SU) Eth LACP Eth6/1(P) Eth6/2(P)
3 Po3(SD) Eth NONE Eth3/3(r)
[root@mw-esx15 ~]# vemcmd show channel type
LTL Channel_Type
------------------
17 MAC Pinning
18 MAC Pinning
76
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
MAC Pinning (Host Pinning Tables) [root@mw-esx15 ~]# vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type
17 Eth3/1 UP UP F/B* 561 0 vmnic0
18 Eth3/2 UP UP F/B* 561 1 vmnic1
49 Veth1 UP UP FWD 0 1 vmk0
[root@mw-esx15 ~]# vemcmd show pc
pce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs
------- ---- ------ --------- ---- ----- -------------- ----
0 1 305 0 N 0 2 17,
1* 3 18,
[root@mw-esx15 ~]# vemcmd show pinning
LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name
10 0 305 32 1 0
12 0 305 32 1 0
49 1c0000a0 305 32 1 0 vmk0
50 1c0000d0 305 32 0 0 vmk1
77
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port Channels – How to Tell Pinning
• Can run from the VSM now
• No need to run command on the VEM
78
n1kv-l3# show int virtual pinning module 5
------------------------------------------------------
Veth Pinned Associated PO List of
Sub Group id interface Eth interface(s)
------------------------------------------------------
Veth2 0 Po5 Eth5/1
Veth4 2 Po5 Eth5/3
Veth5 0 Po5 Eth5/1
Veth6 2 Po5 Eth5/3
Veth7 0 Po5 Eth5/1
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Static Pinning to Sub-Group
• Static Pinning is similar to VMware’s vSwitch active/standby design.
vmk0 VMotion
Sub-group 0 Sub-group 2
Port-channel
C
P
port-profile type ethernet uplink
channel-group auto mode on mac-
pinning relative
port-profile vmkernel
pinning sub-group id 0 backup 2 1
port-profile vmkernel
pinning sub-group id 0 backup 2 1
port-profile type ethernet vmotion
pinning sub-group id 2
vmk0 VMotion
Sub-group 2
Port-channel
C
P
After
failover
Sub-group 1 Sub-group 1
79
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
LACP Port Channels
• Use when single upstream or clustered (vPC,VSS, Catalyst Stack) switch
• Use “channel-group auto mode active” on N1KV
• Use “channel-group # mode active/passive” on upstream switch
• Switchports must be configured with
– spanning-tree portfast trunk
– spanning-tree bpdufilter enable
• Not compatible with Network State Tracking(NST) with LACP
80
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port-Channels - LACP
• LACP allows traffic from each VM to fully utilize multiple links simultaneously.
• Allows faster VMotion and faster VM connectivity by using flow based hashing
port-profile type ethernet uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-10
channel-group auto mode active
no shut
state enable
vSphere
VM VM VM VM
Port-channel
LACP
Upstream switch clustered (vPC,VSS,VBS,Stack…)
81
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
LACP Troubleshooting
• Do not use Network State Tracking(NST) with LACP
• LACP Port-Channel configured on the upstream switches
• Port-profile created with “channel-group auto mode active”
• On the VEM
– vemcmd show lacp
• On the VSM and Upstream Switch
– show port-channel summary
– show lacp counters/neighbor • Are you seeing LACP PDUs?
82
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
LACP Debugging ~ # vemcmd show lacp
LACP Offload is Enabled
---------------------------------------------------
LACP Offload Config for LTL 17
---------------------------------------------------
Channel No : 8
Channel Mode : Active
Port Priority : 0x8000
LACP Bit Set : Yes
SV2# show lacp counters
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
port-channel8
Ethernet10/3 8353 8356 0 0 0 0 0
Ethernet10/1 8353 8356 0 0 0 0 0
83
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
LACP Debugging ~ # vemlog show debug | grep lacp
sflacp ENWID P ( 95) ENW ( 7)
sf_lacp_pdu_utils ENWID P ( 95) ENW ( 7)
sflacp_hostdata ENWID P ( 95) ENW ( 7)
Debug (LTL 16, DIR TX) : Actorstate=7 agg=1 insync=0 coll=0 dis=0 active=1
short_timeout=1 Port ID (0x8000.0x602), Key (7)
Debug (LTL 16, DIR TX) :Partnerstate=2 agg=0 insync=0 coll=0 dis=0 active=0
short_timeout=1 Port ID (0x0.0x0), Key (0)
Debug sf_lacp_tx_pdu_to_upstream: LTL = 18
Debug sf_lacp_tx_pdu_to_upstream, NEW LACP PKT : Src(1), Dst(18), VLAN(1),
FLAGS(1)
[…]
Debug (LTL 18, DIR RX) :Partnerstate=3d agg=1 insync=1 coll=1 dis=1
active=1 short_timeout=0 Port ID (0x8000.0x602), Key (7)
Debug (LTL 16, DIR TX) : Actorstate=3d agg=1 insync=1 coll=1 dis=1 active=1
short_timeout=0 Port ID (0x8000.0x602), Key (7)
84
Virtual Extensible LAN (VXLAN)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Virtual Extensible Local Area Network (VXLAN)
• Ethernet in IP overlay network
– Entire L2 frame encapsulated in UDP (port 4789)
– 50 bytes of overhead
• Include 24-bit VXLAN Identifier
– 16 M logical networks
– Mapped into local bridge domains
– Unique multicast group per segment
• VXLAN can cross Layer 3
• Tunnel between VEMs – VMs do NOT see VXLAN ID
• Egress to Non-VXLAN network
87
Outer
MAC
DA
Outer
MAC
SA
Outer
802.1Q Outer
IP DA
Outer
IP SA
Outer
UDP
VXLAN ID
(24 bits)
Inner
MAC
DA
InnerM
AC
SA
Optional
Inner
802.1Q
Original
Ethernet
Payload CRC
VXLAN Encapsulation Original Ethernet Frame
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Virtual Extensible Local Area Network (VXLAN)
• Each overlay network is known as a VXLAN segment
• Each VXLAN segment identified by a 24-bit segment ID (VNI)
• VXLAN traffic carried between VXLAN Tunnel Endpoints (VTEP)
• VEM module acts as the VTEP
• VM traffic is carried over point to point tunnels between VTEPs
– VM to VM traffic is encapsulated in a VXLAN header
• 1550 MTU for encapsulation overhead
• Encapsulated multicast is always flooded – No IGMP in VXLAN
88
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Deployment Modes: Multicast or Unicast?
• Multicast used to be required for unknown broadcast/unicast on VXLAN
• N1KV 2.2 introduced Unicast Mode and Unicast Mac Distribution Mode
• Multicast (VXLAN 1.0)
– Needs Multicast configured throughout complete network
– IGMP Querier in VLAN
– Multicast routing and proxy ARP across subnets
– VTEPs all join multicast group
– Interoperates with N9K, CSR1K, other Nexus products
• Unicast Mode (VXLAN 1.5)
– VEMs flood each other directly for unknown broadcast/unicast
– Keep a list of other VEMs in each VXLAN
89
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Deployment Modes: When to use MAC Distribution?
• MAC distribution will provide best performance
• No Flooding & Learning
• Full MAC table distributed to each VEM
– VEMs report local MACs to VSM
– VSM distributes {MAC,VTEP} mapping to each VEM
• VXLAN traffic cannot span multiple Nexus 1000V switches*
• Two caveats
– No vEth VXLAN trunk mode support with MAC distribution
– Won’t work with Microsoft NLB
90
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Forwarding Basics
VEM 1 VEM 2
Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn
VEM learns VM’s Source (MAC, Host VXLAN IP) tuple
Broadcast, Multicast, and Unknown Unicast Traffic
VM broadcast & unknown unicast traffic are sent as multicast
Unicast Traffic
Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)
VM VM VM VM
92
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Enhanced VXLAN
VXLAN
(multicast mode)
Enhanced VXLAN
(unicast mode)
Enhanced VXLAN
MAC Distribution
Enhanced VXLAN
ARP Termination
Broadcast /
Multicast
Multicast
Encapsulation
Replication plus
Unicast Encap
Replication plus
Unicast Encap
Replication plus
Unicast Encap
Unknown Unicast
Multicast
Encapsulation
Replication plus
Unicast Encap
Drop Drop
Known Unicast Unicast
Encapsulation Unicast Encap Unicast Encap Unicast Encap
ARP Unicast
Encapsulation
Replication plus
Unicast Encap
Replication plus
Unicast Encap VEM ARP Reply
96
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Configuration: Unicast
• VMkernel interface acts as VTEP
• VSM Control Mode should be L3
• Bridge domain is configured as Unicast or Unicast Mac Distribution
97
feature segmentation
feature vxlan-gateway
port-profile type vethernet vmk-l3-vxlan-vtep
capability l3control
vmware port-group
switchport mode access
switchport access vlan 119
capability vxlan
no shutdown
system vlan 119
state enabled
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Bridge Domain Configuration: Unicast
• Create a bridge-domain in unicast mode
• Scenario 1:
• Scenario 2:
switch(config)# segment mode unicast-only (Global)
switch(config)# bridge-domain segment-cisco
switch(config-bd)# segment id 5000
switch(config-bd)# segment distribution mac
switch(config)# bridge-domain segment-cisco
switch(config-bd)# segment id 5000
switch(config-bd)# segment mode unicast-only (Per BD override)
switch(config-bd)# segment distribution mac
98
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Port-Profile Configuration
• Create an Access Port-Profile with the VXLAN Bridge Domain
• Assign to VM’s in vCenter port-profile type vethernet bd-5000
vmware port-group
switchport mode access
switchport access bridge-domain bd-5000
no shutdown
state enabled
99
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Debugging SV2# show bridge-domain bd-5000
Bridge-domain bd-5000 (2 ports in all)
Segment ID: 5000 (Manual/Active)
Mode: Unicast-only (override)
MAC Distribution: Disable (override)
Group IP: NULL
State: UP Mac learning: Enabled
Veth9, Veth45
SV2# show bridge-domain bd-5000 vteps
Bridge-domain: bd-5000
VTEP Table Version: 21
Port Module VTEP-IP Address VTEP-Flags
---------------------------------------------------------------------------
Veth1 3 14.17.119.21 (D) <---Designated VTEP (vmk)
Veth2 4 14.17.119.22 (D)
Veth13 10 14.17.119.36 (DI) <---VXGW
Veth15 11 14.17.119.36 (DI*)<---VXGW (Standby)
100
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Debugging ~ # vemcmd show vxlan-vteps
Bridge-Domain: bd-5000 Segment ID: 5000
Designated Remote VTEP IPs (*=forwarding publish incapable):
14.17.119.22(DSN: 1), 14.17.119.36(DSN: 1)*
~ # vemcmd show bd bd-name bd-5000
BD 31, vdc 1, segment id 5000, segment group IP 0.0.0.0,
encap VXLAN, vff_mode Anycast,swbd 4096, VLAN 0, 1 ports, "bd-5000"
Segment Mode: Unicast
VTEP DSN: 1 , MAC DSN: 0
Portlist:
52 win2k.eth0
Virtual Machine
in VXLAN 5000
101
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Debugging ~ # vemcmd show l2 segment 5000
Bridge domain 31 brtmax 4096, brtcnt 3, timeout 300
Segment ID 5000, swbd 4096, "bd-5000"
Flags: P - PVLAN S - Secure D - Drop
Type MAC Address LTL timeout PVLAN Remote IP DSN
Dynamic 00:50:56:bc:73:1a 561 121 14.17.119.22 0
Static 00:50:56:a9:00:2e 52 0 0.0.0.0 0
Dynamic 54:7f:ee:2f:33:81 561 2 14.17.119.36 0
ESXi Host #2
VXLAN
Gateway
102
Nexus 1010 and 1110
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSM Deployment Scenarios – Nexus 1110
• VSM on a Nexus 1010/X or 1110-S/X
– It’s still a Virtual Machine
– Up to 14 VSM pairs on one 1110-X cluster
• Always deploy in the appliance pairs!
• N110 allows for Network team to own the virtualization platform
• N110s should go in the Aggregation Layer
• Stretched Model requires
– L2 Connectivity
– 10ms latency
Cisco Cloud Services Platform
104 *Next Release
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
1110-S/X Deployment Scenario
105
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Cloud Services Platform (CSP)
• Based off UCS C2x0 M3 server
– Same CIMC/BIOS/firmware
– Provide 6 x 1G network connections
– 1110-X 2 x 10G - SP1(7) • 10G available only on purchase. No upgrade available.
– Encryption Accelerator Card for Citrix VPX – SP1(7)
• Virtual Service Blade (VSB) Support
– 1010/1110-S supports up to 10
– 1010/1110-X supports up to 14
Nexus 1010/1010-X/1110-S/1110-X
106
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Cloud Services Platform (CSP)
• Current supported VSBs
– Nexus 1000V VSM (ESX/HyperV/KVM)
– Virtual Security Gateway (VSG)
– Network Analysis Module (NAM)
– Data Center Network Manager (DCNM)
– Citrix NetScaler VPX
107
VSB Minimum Version
HyperV SP(6.1)
VXLAN GW SP1(6.1)
Citrix Netscaler SP1(6.2)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Cisco Cloud Services Platform (CSP)
• Must be deployed in pairs
– No option for standalone
• Deploy in the Aggregation Layer
• Must be in the same L2 domain for management and control
• Can be geographically diverse
• Uses same HA mechanism as VSM with domain-id and control vlan
– Do not overlap the domain-id between a 1x10 and a VSM
• What’s not supported?
– Primary and Secondary VSM on same 1x10
– Primary VSM on ESX and Secondary VSM on 1x10 or vice versa
108
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VSB Backups using Import/Export
• Works with VSM, NAM, and VSG
• Can Import/Export both primary and secondary
• Export requires that VSB be shutdown
• Images are stored in “export-import/” dir on bootflash
– Can be manually copied off to remote storage
109
n1010-1# copy scp://[email protected]/root/Vdisk4.img.tar.00
bootflash:export-import vrf management
n1010-1(config)# virtual-service-blade training
n1010-1(config-vsb-config)# import primary Vdisk4.img.tar.00
Note: import started..
Note: please be patient..
Note: Import cli returns check VSB status for completion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Network Classes and Topologies
• Management
– Carries the mgmt0 interface of the 1x10
– Carries the mgmt0 traffic for all VSMs installed
• Control
– Carries all the control and packet traffic for the VSMs installed on the 1x10
– Carries control traffic for HA between primary and secondary 1x10
• Data
– Used by Virtual Service Blades other than VSM
• Passthrough
– Binds physical NIC to VSB
• 5 Network Topologies choices
110
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Network Topologies
Uplink Type Management VLAN Control VLAN Data VLAN
1 Ports 1 and 2 Ports 1 and 2 Ports 1 and 2
2 Ports 1 and 2 Ports 1 and 2 (HA) Ports 3-6 (LACP)
3 Ports 1 and 2 Ports 3-6 (LACP) Ports 3-6 (LACP)
4 Ports 1 and 2 Ports 3 and 4 Ports 5 and 6
Flexible There is no traffic segregation based on traffic class.
*Must use for VXGW deployements.
111
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Recommendations
• If you are not planning on using other VSBs
– Topology 3 gives best bandwidth and redundancy for control VLAN • Negative is that is harder to configure
• If using VXGW, Netscaler, or shared between production / lab network
– Topology 5 is Flexible
• Flexible allows any configuration
– Recommend port-channels
– Remember VSM latency is key over bandwidth
• Use VPC or VSS upstream if you have it
112
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Participate in the “My Favorite Speaker” Contest
• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)
• Send a tweet and include
– Your favorite speaker’s Twitter handle @juicyUCS
– Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Promote Your Favorite Speaker and You Could be a Winner
113
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Complete Your Online Session Evaluation
• Give us your feedback and you could win fabulous prizes. Winners announced daily.
• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
114
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics – Moscone Center West – 3rd Floor Lobby
– Discuss “Experiences with Cisco Services” with Distinguished Service Engineers
• Meet the Engineer 1:1 meetings
115
Appendix A – L2 Troubleshooting
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
L2 Control VEM – VSM Troubleshooting Steps
1. VSM MAC address
2. VSM is connected to vCenter
3. VSM has Control VLAN on right interface
4. Uplink port-profile has Control vlan
5. VEM sees control VLAN
6. VEM and VSM see each others MAC
7. Physical network sees VEM and VSM MAC
8. VSM sees heartbeat messages from VEM
119
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
n1kv-l2# show svs neighbors
Active Domain ID: 422
AIPC Interface MAC: 0050-56a9-2535
Inband Interface MAC: 0050-56a9-2537
Step 1: VSM MAC
• Need for L2 troubleshooting
• On VSM run show svs neighbors
• Its the AIPC Interface MAC
120
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
n1kv-l2# show svs connections
connection VC:
ip address: 172.18.217.241
remote port: 80
protocol: vmware-vim https
certificate: default
datacenter name: Harrington
admin:
max-ports: 8192
DVS uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e
config status: Enabled
operational status: Connected
Step 2: VSM – vCenter Connectivity
• Verify VSM is connected to vCenter
121
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Step 3: Verify VSM VM Control interface
• 1st interface listed is Control Interface
• Interface connected?
122
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Step 4: Verify Uplink Port-Profile
• The first ESX interface added to the N1KV must have Control VLAN
• Verify uplink port-profile has Control VLAN defined and system VLAN
123
n1kv-l2# show run port-profile uplink version 4.2(1)SV1(5.1) port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-3967,4048-4093 no shutdown system vlan 2 state enabled
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Step 5: Verify VEM Sees Control VLAN
• Verify VEM sees control VLAN with commands
– vemcmd show card
– vemcmd show port
– vemcmd show trunk
124
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
[~ # vemcmd show card
Card UUID type 2: 33393138-3335-5553-4537-31314e343636
Card name: cae-esx-154
Switch name: n1kv-l2
Switch alias: DvsPortset-0
Switch uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e
Card domain: 422
Card slot: 5
VEM Tunnel Mode: L2 Mode
VEM Control (AIPC) MAC: 00:02:3d:11:a6:04
VEM Packet (Inband) MAC: 00:02:3d:21:a6:04
VEM Control Agent (DPA) MAC: 00:02:3d:41:a6:04
..
..
Card control VLAN: 2
Card packet VLAN: 2
Vemcmd show card
• Control, packet vlans and domain-ID match with VSM
125
MAC the VSM
should learn for
VEM
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
~ # vemcmd show port-old
LTL IfIndex Vlan/ Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name
SegId
6 0 1 T 0 32 32 VIRT UP UP 1 Trunk vns
8 0 3969 0 32 32 VIRT UP UP 1 Access
9 0 3969 0 32 32 VIRT UP UP 1 Access
10 0 2 0 32 32 VIRT UP UP 1 Access
11 0 3968 0 32 32 VIRT UP UP 1 Access
12 0 2 0 32 32 VIRT UP UP 1 Access
13 0 1 0 32 32 VIRT UP UP 0 Access
14 0 3971 0 32 32 VIRT UP UP 1 Access
15 0 3971 0 32 32 VIRT UP UP 1 Access
16 0 1 T 0 32 32 VIRT UP UP 1 Trunk ar
17 25010000 1 T 0 32 32 PHYS UP UP 1 Trunk vmnic0
Vemcmd show port-old
• Ports with LTLs 8, 9,10 are UP and CBL states are 1.
• ESX Physical ports are UP and CBL states 1.
126
Local Target Logic (LTL) is an index to address a port, or group of ports. Data path lookup engine takes LTL
as input, and gives LTL as output.
LTL scheme: [0-14: internal ports] [15-271: pNICs,VMs, etc…]
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
~ # vemcmd show trunk
Trunk port 6 native_vlan 1 CBL 1
vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1,
vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1,
vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,
Trunk port 16 native_vlan 1 CBL 1
vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1,
vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1,
vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,
Trunk port 17 native_vlan 1 CBL 1
vlan(1) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1,
vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,
Vemcmd show trunk
• Control and packet are CBL states 1 on the physical ports.
127
~ # vemcmd show port vlans
Native VLAN Allowed
LTL VSM Port Mode VLAN State Vlans
17 Eth5/1 T 1 FWD 2,10-11,150-155
~ #
vemcmd show port vlans
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
~ # vemcmd show l2 2
Bridge domain 9 brtmax 4096, brtcnt 32, timeout 300
VLAN 2, swbd 2, ""
Flags: P - PVLAN S - Secure D - Drop
Type MAC Address LTL timeout Flags PVLAN
Static 00:02:3d:21:a6:04 12 0
Dynamic 00:50:56:a9:25:35 17 1
Step 6: VEM and VSM See Each Other’s MAC
• Is the VEM learning the MAC of the VSM?
• On VEM “vemcmd show l2 <control-vlan>” do you see the mac of the VSM?
128
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
n1kv-l2# show mac address-table vlan 2
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---
2 0002.3d21.a604 static 0 N1KV Internal Port 5
2 0002.3d41.a604 static 0 N1KV Internal Port 5
VEM and VSM See Each Other’s MAC
• Is the VSM learning the MAC of the VEM?
129
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
cae-cat6k-1#show mac-address-table vlan 2
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
* 2 0050.5677.7770 dynamic Yes 360 Gi3/48
* 2 0050.56a9.2535 dynamic Yes 0 Gi4/9
* 2 3333.0000.0016 static Yes - Switch,Stby-Switch
* 2 0002.3d41.a604 dynamic Yes 0 Gi1/4
Step 7: Physical Switch Mac Table
• Check the physical switch MAC address table
• Are the MACs of the VEM and VSM getting learned by the physical switches in the right VLANs?
130
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Step 8: VEM – VSM Heartbeat
• One Heartbeat per second per VEM from VSM
• Timeout for VEM from VSM is 6 seconds of missed heartbeats
• After 6 seconds VSM will drop VEM
• Use vempkt capture to view heartbeats
• SPAN physical switch ports for heartbeats
131
Appendix B – Miscellaneous Commands
Appendix C – VXLAN Multicast
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Configuration: Multicast
• VMkernel interface to act as VTEP
• VSM Control Mode should be L3
• Multicast for Broadcast traffic
• IP Multicast forwarding is required
– Multicast addresses
– Multiple segments can be mapped to a single multicast group
– If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN
– If VXLAN transport is traversing routers • Multicast routing must be enabled.
• Proxy ARP must also be enabled
• 1550 MTU for VXLAN encapsulation overhead
134
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Configuration: Multicast
• Upstream Switch Configuration
– Enable IGMP Querier
– Set physical switch port MTU to 1550
– Enable proxy-arp on upstream SVI
• ESXi Host
– Create VMK interface for VXLAN
• Nexus 1000V
– Enable “feature segmentation”
– Create a Bridge Domain
– Create a port-profile for VTEP VMK interface
– Create a veth port-profile for the VMs
135
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Configuration: Multicast
• Increase the MTU on your eth port-profile
n1kv-l3(config)# port-profile type eth uplink
n1kv-l3(config-port-prof)# mtu 1550
• Create veth port-profile for VXLAN VMK interface
n1kv-l3(config)# port-profile type vethernet VXLAN-VMK
n1kv-l3(config-port-prof)# switchport mode access
n1kv-l3(config-port-prof)# switchport access vlan 11
n1kv-l3(config-port-prof)# no shutdown
n1kv-l3(config-port-prof)# system vlan 11
n1kv-l3(config-port-prof)# vmware port-group
n1kv-l3(config-port-prof)# capability vxlan
n1kv-l3(config-port-prof)# state enabled
136
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Configuration: Multicast
• Configure the Bridge Domain
Maps a segment ID to a multicast address
Segment ID >4096
n1kv-l3(config)# bridge-domain vxlan-1
n1kv-l3(config-bd)# segment id 5000
n1kv-l3(config-bd)# group 224.3.5.2
• Create VM port-profile
n1kv-l3(config)# port-profile type veth vm-vxlan-1
n1kv-l3(config-port-prof)# vmware port-group
n1kv-l3(config-port-prof)# switchport mode access
n1kv-l3(config-port-prof)# switchport access bridge-domain vxlan-1
n1kv-l3(config-port-prof)# no shut
n1kv-l3(config-port-prof)# state enabled
–
137
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Troubleshooting Tips
• Verify your Bridge Domains, VM port-profiles, and VXLAN VMK port-profiles
• Verify multicast on your upstream switches
– show ip igmp snooping
– Do you see the VTEPs
• Use vmkping on the ESXi host to verify network and MTU
– Use 1542 to cover the addition of the ICMP header
– ~ # vmkping -s 1542 -d 1.1.1.1
• Verify the VEM has the right VXLAN capability
~ # vemcmd show vxlan interfaces
LTL IP
---------------
69 1.1.1.2
138
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Troubleshooting Tips
~ # vemcmd show port vlans
LTL VSM Port Mode VLAN/ State Vlans/SegID
17 Eth4/1 T 1 FWD 25,626-640
18 Eth4/2 T 1 FWD 25,626-640
53 Veth19 A 6000 FWD 6000
• Verify the VEM was programmed correctly
~ # vemcmd show segment 6000
BD 23, vdc 1, segment id 6000, segment group IP 225.6.26.10, swbd 4096, 2 ports, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"
Portlist:
53 vse-vCDNI-6-26-vl634-backed (b6
68 vCDNI-2 (5ac7d73c-d1d1-4877-8ef
139
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• vemcmd show port
• vemcmd show igmp <vlan>
• vemcmd show l2 segment <segment-id>
• vemcmd show vxlan-encap [ltl/mac] <ltl/MAC address>
• vemcmd show vlxan-stats all
• Detailed slides in the Appendix
140
Appendix D - Additional VXLAN TShoot
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• Verify Multicast Upstream Nexus 7K/5K
– Verify querier is enabled for vlan VMK interfaces are on switch# show run
vlan configuration 634
ip igmp snooping querier 1.1.1.161
142
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN other Useful Commands
• Verify IGMP snooping is configured
CWD.35.04-7000-1# show ip igmp snooping vlan 634
IGMP Snooping information for vlan 634
IGMP snooping enabled
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 1.1.1.161, version: 3, i/f Po1
Querier interval: 125 secs
Querier last member query interval: 1 secs
Querier robustness: 2
Switch-querier enabled, address 1.1.1.161, currently running
…..
IGMPv3 Report suppression disabled
Link Local Groups suppression disabled
Router port detection using PIM Hellos, IGMP Queries
Number of router-ports: 1
Number of groups: 1
VLAN vPC function enabled
Active ports:
Po1 Po9 Po17 Po25
Po31 Po52 Po100 Eth2/30
143
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLN other Useful Commands
• Verify multicast IP address for the VXLAN is being learned
CWD.35.04-7000-1# show ip igmp snooping groups vlan 634
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core
port
Vlan Group Address Ver Type Port list
634 */* - R Po1
634 225.6.26.10 v2 D Po100 Eth2/30
144
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• vemcmd show port
– Will show ports that are on a vxlan ~ # vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type
17 Eth4/1 UP UP F/B* 305 0 vmnic0
18 Eth4/2 UP UP F/B* 305 1 vmnic1
49 Veth2 UP UP FWD 0 0 vmk0
...
53 Veth19 UP UP FWD 0 vse-vCDNI-6-26-vl634-backed (b6
54 Veth16 UP UP FWD 0 0 vse-vCDNI-6-26-vl634-backed (b6
...
68 Veth21 UP UP FWD 0 vCDNI-2 (5ac7d73c-d1d1-4877-8ef
69 Veth22 UP UP FWD 0 0 vmk1 vxlan
305 Po2 UP UP F/B* 0
145
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• vemcmd show igmp <vlan>
– Verify that multicast is enabled ~ # vemcmd show igmp 634
IGMP is ENABLED on VLAN 634
Multicast Group Table:
Group */*, Multicast LTL: 4410
• vemcmd show l2 segment <segment-id>
– Verify the VEM is learning MAC addresses in the VXLAN ~ # vemcmd show l2 segment 6000
Bridge domain 23 brtmax 4096, brtcnt 3, timeout 300
Segment ID 6000, swbd 4096, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"
Flags: P - PVLAN S - Secure D - Drop
Type MAC Address LTL timeout Flags PVLAN Remote IP
Static 00:50:56:01:02:0a 68 0 0.0.0.0
Dynamic 00:50:56:01:02:09 305 1 1.1.1.1
Static 00:50:56:01:02:15 53 0 0.0.0.0
146
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• vemcmd show vxlan-encap [ltl/mac] <ltl/MAC address>
– Identify the traffic path a MAC or LTL will utilize ~ # vemcmd show vxlan-encap ltl 68
Encapsulation details for LTL 68 in BD "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18":
Source MAC: 00:50:56:01:02:0a
Segment ID: 6000
Multicast Group IP: 225.6.26.10
Encapsulating L2 LISP Interface LTL: 69
Encapsulating Source IP: 1.1.1.2
Encapsulating Source MAC: 00:50:56:7e:0e:b6
Pinning of L2 LISP Interface to the Uplink:
LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name
69 1c000150 305 32 0 0 vmk1
147
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Other Useful Commands
• vemcmd show vlxan-stats all
– Show VXLAN traffic stats ~ # vemcmd show vxlan-stats all
LTL Ucast Mcast Ucast Mcast Total
Encaps Encaps Decaps Decaps Drops
53 67 300 47 0 23
68 11701 47135 11690 61 12
69 11768 125793 11737 61 0
148
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
VXLAN Load Balancing
• With LACP port-channel 5-tuple hash is used
– Use singe VMK VXLAN interface
– VEM does the hashing across all the links
– Remember to change load balancing to 5-tuple hashing • On the upstream switch and on the VSM
• With VPC MAC Pinning
– Create a VMK VXLAN interface for each available uplink
– VEM will pin an interface to each available link
– The VEM will distribute the VM's flows between the vmknics based on a hash of the source MAC.
149
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Verification: Unicast
Verify the bridge-domain configuration on VSM switch# sho bridge-domain
Global Configuration:
Mode: Unicast-only
MAC Distribution: Disable
Bridge-domain segment-cisco (3 ports in all)
Segment ID: 9001 (Manual/Active)
Mode: Unicast-only (default)
MAC Distribution: Disable (default)
Group IP: NULL
State: UP Mac learning: Enabled
Veth2, Veth3, Veth5
Verify the bridge-domain configuration on VEM
switch# module vem 4 execute vemcmd show bd bd-name segment-cisco
BD 26, vdc 1, segment id 9001, segment group IP 0.0.0.0, swbd 4102, 2 ports, "segment-cisco"
Segment Mode: Unicast
VTEP DSN: 1 , MAC DSN: 1
Portlist:
53 RedHat_VM1_112.eth4
54 RedHat_VM1_112.eth5
~ #
If MAC Distribution is enabled this will be ‘Enable’
If MAC Distribution is enabled this will be
“Segment Mode: Unicast, Mac-Distribution”
VTEP and MAC download sequence numbers should
be checked against VTEP entries (vemcmd show
vxlan-vteps) and MAC entries (vemcmd show l2 bd
bd-name <>) respectively
150
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Verification (Port configuration)
Verify the port configuration on VSM switch# sho int switchport | begin Vethernet2
Name: Vethernet2
Switchport: Enabled
Switchport Monitor: Not enabled
Operational Mode: access
Access Mode VLAN: 0 (none)
Access BD name: segment-cisco
[SNIP]
151
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Verification (Port configuration)
switch# module vem 4 execute vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type
17 Eth4/1 UP UP F/B* 561 0 vmnic0
49 DOWN UP BLK 0 RedHat_VM1_112 ethernet7
50 Veth8 DOWN UP BLK 0 RedHat_VM1_112.eth8
51 Veth4 UP UP FWD 0 0 vmk1 VXLAN
52 DOWN UP BLK 0 RedHat_VM1_112.eth6
53 Veth2 UP UP FWD 0 RedHat_VM1_112.eth4
54 Veth3 UP UP FWD 0 RedHat_VM1_112.eth5
561 Po2 UP UP F/B* 0
* F/B: Port is BLOCKED on some of the vlans.
One or more vlans are either not created or
not in the list of allowed vlans for this port.
Please run "vemcmd show port vlans" to see the details.
~ #
Verify the port configuration on VEM
152
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Verification (VTEP Distribution)
Verify the VTEP distribution on VEM
switch# sho bridge-domain segment-cisco vteps
D: Designated VTEP I:Forwarding Publish Incapable VTEP
Bridge-domain: segment-cisco
VTEP Table Version: 2
Ifindex Module VTEP-IP Address
-----------------------------------------------------------------
-------------
Veth4 4 10.106.199.116(D)
Veth1 5 10.106.199.117(D)
switch#
Verify the VTEP distribution on VSM
switch# module vem 4 execute vemcmd show vxlan-vteps
Bridge-Domain: segment-cisco Segment ID: 9001
Designated Remote VTEP IPs (*=forwarding publish incapable):
10.106.199.117(DSN: 1),
To be compared with
echo “show vxlan
version-table” output on
VEM
Compare against “vemcmd show bd bd-name <>”
VTEP DSN output
153
© 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public
Verification (MAC table in unicast only mode)
switch# module vem 4 execute vemcmd show l2 bd-name segment-cisco
Bridge domain 26 brtmax 4096, brtcnt 3, timeout 300
Segment ID 9001, swbd 4102, "segment-cisco"
Flags: P - PVLAN S - Secure D - Drop
Type MAC Address LTL timeout Flags PVLAN Remote IP DSN
Dynamic 00:50:56:83:01:4e 561 1 10.106.199.117 0
Static 00:50:56:83:01:61 54 0 0.0.0.0 0
Static 00:50:56:83:01:60 53 0 0.0.0.0 0
switch#
MAC address table will display remote IP learning in the segment-cisco bridge domain
154