HyperFlex Configuration Best Practices and...

HyperFlex Configuration Best Practices and Troubleshooting Scenarios

Clifford Aldan, Customer Support Engineer, Cisco Global TACJonathan Gorlin, Solutions Architect, HyperFlex

BRKVIR-2700

Abstract

HyperFlex is Cisco's HyperConverged Infrastructure solution designed to meet today's Virtual Desktop Infrastructure and Virtual Server Infrastructure workloads. This session enables professionals administering the datacenter to understand best practices when configuring HyperFlex and troubleshoot common scenarios.

BRKVIR-2700 3

The main goals of this session

• Configurations in key areas of the HyperFlex solution – Northbound Switch, UCS, vSphere, and the HX Data Platform (HXDP)

• Common troubleshooting scenarios experienced on HyperFlex

BRKVIR-2700 4

This session does not include

• HX performance deep dive

• Replication, backup, and DR scenarios

• VDI configuration and deployment

• Distributed switch configuration (Nexus 1000v)

• HX 2.0

• Roadmap

BRKVIR-2700 5

Related Cisco Live Sessions

Session-ID Date and Time Session Name

BRKCOM-2004 Thursday, 11:30 AM Cisco HyperFlex Deep Dive and Performance

BRKINI-4003 Tuesday, 11:15 AM What's New with Cisco HyperFlex Systems 2

PSODCN-2375 Wednesday, 1:15 PM Panel: What’s New with HyperFlex All Flash Release and Learn How ForePaaS Built a Hybrid

Cloud Service on HyperFlex

LTRDCN-1700 Tuesday, 9:30 AM Deployment and operation of Cisco Hyperflex HX Data Platform

BRKGS-2004 Wednesday, 12:35 PM Learn How to enable your Always-On Enterprise with Cisco HyperFlex and Veeam!

BRKGS-1344 Wednesday, 2:10 PM Increase productivity, consolidate data management, and ensure business continuity with

Commvault and Cisco UCS

BRKPLT-2000 Tuesday, 4:15 PM Enabling Secure Delivery of Apps and Data with Superior Performance and Agility via Citrix

and Cisco

BRKVIR-2700 6

and Cisco

BRKVIR-2700 7

Cisco SparkAsk Questions, Get Answers, Continue the Experience

Use Cisco Spark to communicate with the Speaker and fellow participants after the session

Download the Cisco Spark app from iTunes or Google Play

1. Go to the Cisco Events mobile app

2. Find this session

3. Click the Spark button under Speakers in the session description

4. Enter the room, room name = BRKVIR-2700

5. Join the conversation!

The Spark Room will be open for 2 weeks after Cisco Live

BRKVIR-2700 8

• Prerequisites

• Northbound Switch

• UCS

• vSphere

• HX Data Platform

• Troubleshooting Scenarios

Agenda

Prerequisites

Have a planning call with all parties that will be involved in the HX deployment

Server Team

Network Team

Storage Team

Security Team

Cisco or Partner

UCS, VMware, HX

Switches

Routers

FC/FCoE Switches

External SAN/NAS

Account Team

TAC Support

Firewalls

Security Gateways

HX HCI

Initiative

BRKVIR-2700 11

Fill out the latest version of the HX pre-install checklist

BRKVIR-2700 12

Fill out the latest version of the HX pre-install checklist

• UCS Information

• VLANs

• IP Addresses (ESXi, SCVM)

• vSphere Information (vCenter, ESXi, Datacenter name, Cluster name)

• HXDP Information

• DNS, NTP, Time Zone

• Auto Support

BRKVIR-2700 13

Review the interoperability requirements to ensure compatibility between hardware and software

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2

Enterprise Plus, Standard,

Essentials Plus, ROBO

2.2(6f) – 1.7.1

2.2(7c) – 1.7.1-14835

HyperFlex

1.7.1, 1.7.1-14835

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2

2.2(7c)

HyperFlex

ESXi 5.5 U3 (HX240 only)

Enterprise Plus

6.0 U1b, U2, U2 P3

3.1(2b) – 1.8.1a, 1.8.1b,

1.8.1c, 1.8.1e

3.1(2c) – 1.8.1c, 1.8.1e

HyperFlex

1.8.1a, 1.8.1b, 1.8.1c, 1.8.1e

BRKVIR-2700 14

Review the interoperability requirements to ensure compatibility between hardware and software

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2

2.2(6f) – 1.7.1

2.2(7c) – 1.7.1-14835

HyperFlex

1.7.1, 1.7.1-14835

ESXi 5.5 U3

Enterprise Plus

6.0 U1b, U2

2.2(7c)

HyperFlex

ESXi 5.5 U3 (HX240 only)

Enterprise Plus

6.0 U1b, U2, U2 P3

3.1(2b) – 1.8.1a, 1.8.1b,

1.8.1c, 1.8.1e

3.1(2c) – 1.8.1c, 1.8.1e

HyperFlex

1.8.1a, 1.8.1b, 1.8.1c, 1.8.1e

BRKVIR-2700 15

Validate Network Services are stable, reachable, and continuous (for the lifetime of the cluster)

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HyperFlex Cluster

BRKVIR-2700 16

Allow required ports through the firewall

BRKVIR-2700 17

HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

INBOUND

BRKVIR-2700 18

HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

SMTP DNS NTP SNMP

25 53 123 162

TCPTCP

UDPTCP

INBOUND OUTBOUND

BRKVIR-2700 19

HTTP HTTPS KVM SNMP

80 443 2068 161

TCP TCP TCPSNMP

SMTP DNS NTP SNMP

25 53 123 162

TCPTCP

UDPTCP

SSH POP3 IMAP4VCENTER

VCENTER

INSTALLER

(993)902 9443 8089

TCP TCP TCP TCP/UDP TCP TCP

INBOUND OUTBOUND

INBOUND/OUTBOUND

BRKVIR-2700 20

• Prerequisites

• UCS

• vSphere

Agenda

Northbound Switch

Assess your northbound switches for features, licenses, port count, port config, cable requirements

What features need to be enabled/licensed?

Do I have enough ports/port licenses?

What’s the switchport configuration?

BRKVIR-2700 23

BRKVIR-2700 24

BRKVIR-2700 25

BRKVIR-2700 26

Create VLANs for the following network traffic types

•Used for all VM traffic

•L3 routable

• VM live migrations

•L2 or L3

•ESXi/SCVM storage traffic

•HXDP replication

•L2 Non-routed

•vCenter, ESXi, SCVM communication

•L3 Routable

Management Storage Data

VM NetworkvMotion

OPTIONAL

External

Storage

iSCSI, NFS,

BRKVIR-2700 27

Use VLANs other than VLAN 1

Default VLAN in which all switch control plane traffic use – telnet, SNMP, syslog

Protocol communication between switches (i.e., CDP, STP, VTP, etc)

All switchports are part of VLAN 1

Will cause networking issues, especially for disjoint Layer 2 (i.e., DMZ)

CONTROL PLANE

BRKVIR-2700 28

Set switchports to UCS FIs as trunk ports to allow multiple VLANs to traverse each uplink

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

switch# configure terminal

switch(config)# interface port-channel 201

switch(config-if)# switchport mode trunk

switch(config-if)# switchport trunk allowed vlan

100, 200, 300, 400

BRKVIR-2700 29

Set switchports connecting to UCS FIs as edge ports to support End Host Mode

FIs seen as a host with a large number of NICs and MAC addressesEth1/1

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Northbound Switch

BRKVIR-2700 30

Set switchports connecting to UCS FIs as edge ports to support End Host Mode

For Cisco NX-OS switchports to UCS FIs switch# configure terminal

switch(config-if)# spanning-tree port type edge trunk

For Cisco IOS switchports to UCS FIs switch# configure terminal

switch(config-if)# spanning-tree portfast edge trunk (IOS release 12.2(33)SXI and later)

switch(config-if)# spanning-tree portfast trunk

(IOS releases prior to 12.2(33)SXI)

BRKVIR-2700 31

Enable jumbo frames for interfaces connecting to the FIs

• For Cisco NX-OS, use a policy-map and system qospolicy-map type network-qos FCoE-with-Jumbo

class type network-qos class-fcoe

pause no-drop

mtu 2158

class type network-qos class-default

mtu 9216

multicast-optimize

system qos

service-policy type network-qos FCoE-with-Jumbo

• For Cisco IOS, configure at interface level6500(config)# interface gigabitEthernet 1/1

6500(config-if)# mtu ?

<1500-9216> MTU size in bytes

6500(config-if)# mtu 9216

Jumbo Frame (9216 MTU)

BRKVIR-2700 32

Enable jumbo frames end-to-end to support Storage Data and vMotion traffic

Policy Map

Switchport

QoSSystem Class

vNIC vSwitchVMkernelAdapter

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

HX240cM4

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 33

Enable jumbo frames end-to-end to support Storage Data and vMotion traffic

Policy Map

Switchport

QoSSystem Class

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

HX240cM4

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 34

Map the CoS values to the CoS settings of the UCS QoS System Class

• For Cisco NX-OSswitch# configure terminal

switch(config)# class-map type qos class_platinum

switch(config-cmap-qos)# match cos 5

• For Cisco IOSswitch# configure terminal

switch(config)# class-map class_bronze

switch(config-cmap)# match cos 1

Platinum

Silver

Bronze

Platinum

Silver

Bronze

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

BRKVIR-2700 35

Configure port-channel for redundancy, wider bandwidth, and load balancing

BRKVIR-2700 36

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

BRKVIR-2700 37

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

vPC Non-vPC

BRKVIR-2700 38

Sample configuration: vPC with 4 links from each Fabric Interconnect Switch 1 Configuration

interface Ethernet1/25-26

description “Link to HX-UCS-FI-A Eth1/25-26”

channel-group 201 mode active

interface port-channel201

description "vPC to HX-UCS-FI-A"

switchport mode trunk

switchport trunk allowed vlan 100,200,300,400

spanning-tree port type edge trunk

speed 10000

vpc 201

description “Link to HX-UCS-FI-B Eth1/27-28”

description "vPC to HX-UCS-FI-B"

speed 10000

vpc 202

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Switch 1 Switch 2

BRKVIR-2700 39

Sample configuration: vPC with 4 links from each Fabric Interconnect Switch 2 Configuration

description "vPC to HX-UCS-FI-B"

speed 10000

vpc 202

description "vPC to HX-UCS-FI-A"

speed 10000

vpc 201

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Switch 2Switch 1

BRKVIR-2700 40

Sample configuration: port-channel with 4 links from each Fabric Interconnect

Switch 1 Configuration interface Ethernet1/25-26

description "PC to HX-UCS-FI-A Eth1/25-26"

speed 10000

description "PC to HX-UCS-FI-B Eth1/27-28"

speed 10000

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Switch 1 Switch 2

Non-vPC

BRKVIR-2700 41

Sample configuration: port-channel with 4 links from each Fabric Interconnect

Switch 2 Configuration interface Ethernet1/25-26

description "PC to HX-UCS-FI-B Eth1/25-26"

speed 10000

description "PC to HX-UCS-FI-A Eth1/27-28"

speed 10000

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Switch 2Switch 1

Non-vPC

BRKVIR-2700 42

OPTIONAL – Set up FC/FCoE ports to connect HyperFlex to an external FC/FCoE SAN

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

7 8HX220c

Console!

HyperFlex

7 8HX220c

Console!

HyperFlex

7 8HX220c

Console!

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

FC SAN

FC Only Switches

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

7 8HX220c

Console!

HyperFlex

7 8HX220c

Console!

HyperFlex

7 8HX220c

Console!

HyperFlex

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

FC/FCoESANSwitches

with Unified Ports –EthernetFC/FCoE

HyperFlex

BRKVIR-2700 43

• Prerequisites

• UCS

• vSphere

Agenda

Set up Call Home to get email alerts from UCS Manager (UCSM)

Refer to the Cisco UCS Manager

System Monitoring Guide,

Release 3.1

(OPTIONAL) Fill out this section if

using Smart Call Home (SCH)

• Contact your Cisco Account

team to obtain information

BRKVIR-2700 46

IF NEEDED – Use Auto-Install to upgrade/downgrade the UCS firmware (infra/server)

BRKVIR-2700 47

Make sure all UCS components are at the correct version and the Activate Status is in “Ready” state

BRKVIR-2700 48

Configure unified ports – network uplink, server, or FC uplink ports

BRKVIR-2700 49

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Network Uplink ports

BRKVIR-2700 50

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI BFI A

770W AC

ACT LINK ACT LINK

Server ports

BRKVIR-2700 51

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI B

1400W AC

1400W ACM

ACT LINK ACT LINK

PSUMLOM

Server ports

BRKVIR-2700 52

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

FAN STATUS

CHS A56

FAN 1 FAN 5 FAN 2 FAN 6 FAN 3 FAN 7 FAN 4 FAN 8

UCS 2204XP

Server ports

FI BFI A

BRKVIR-2700 53

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

FC Uplink ports

BRKVIR-2700 54

When converting to FC uplinks, use the Configure Unified Ports wizard (FI reboot required)

BRKVIR-2700 55

When converting to FC uplinks, use the Configure Unified Ports wizard (FI reboot required)

BRKVIR-2700 56

When considering VLANs to create, note that the HX installer will create certain VLANs in UCSM

Can be a comma separated list of guest

BRKVIR-2700 57

2000, 4000, 67,

5, 4, 1000, 500,

6, 3000

Create the remaining VLANs to match those configured on the northbound switches

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

BRKVIR-2700 58

When deploying disjoint Layer 2, prune the VLANs to the corresponding network uplinks

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Production

10, 20

FI A FI BVLAN 1 allowed on all ports

VLAN 1 cannot be pruned off the FIs

BRKVIR-2700 59

VLAN 1

listed!

BRKVIR-2700 60

• Do this on both FIs

BRKVIR-2700 61

• Do this on both FIs

• Assign all VLANs to their respective uplinks

BRKVIR-2700 62

HX installer configures QoS and sets Jumbo Frames for Storage and vMotion networks

• Refer to this table when configuring CoS value (1, 2, 4, 5) on the northbound switches

BRKVIR-2700 63

HX installer configures QoS and sets Jumbo Frames for Storage and vMotion networks

• Refer to this table when configuring CoS value (1, 2, 4, 5) on the northbound switches

• While HyperFlex initial configuration does not use Priority Best Effort, it can be used for other traffic types

BRKVIR-2700 64

Eight updating vNIC templates are created and assigned a QoS policy, MTU size, and a VLAN*

Silver

hv-mgmt-a

hv-mgmt-b

MTU 1500

VLAN hx-inband-

vm-network-a

vm-network-b

MTU 1500

VLAN vm-network

Bronze

hv-vmotion-a

hv-vmotion-b

MTU 9000

VLAN hx-vmotion

Platinum

storage-data-a

storage-data-b

MTU 9000

VLAN hx-storage-

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

BRKVIR-2700 65

Policy Map

Uplink Interface

QoSSystem Class

Enable jumbo frames end-to-end to support HX storage and vMotion traffic

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

HX240cM4

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 66

OPTIONAL: Create additional vNIC templates and MAC pools for other traffic types

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

BRKVIR-2700 67

OPTIONAL: Create vHBA templates and WWNN/WWPN pools for FC/FCoE SAN connection

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

FC/FCoE SAN

FC/FCoE Switches

BRKVIR-2700 68

For compute nodes, configure local disk config and boot policies if ESXi is not installed on SD cards

BRKVIR-2700 69

• Prerequisites

• UCS

• vSphere

Agenda

vSphere

Deploy vCenter outside of the HX cluster if one is not present

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HyperFlex VDI/VSI Cluster

vCenter

Virtual Server Farm

DNS vCenter

BRKVIR-2700 72

If a vCenter server is already present, double check that it meets the HX interoperability

vCenter

• vCenter version must be equal to or higher than the ESXi version on the HX node

• vCenter meets the requirements to manage all the Hosts and VMs in the environment – Tiny, Small, Medium, Large

• TCP ports used by HX are NOT used by other 3rd party software or plugins (i.e., port 8089, port 80 for vCenter on Windows)

BRKVIR-2700 73

Use the local SSO account during the HX installation

administrator@vsphere.local

• Local SSO account has root-level privileges to vCenter

• It’s a member of the SystemConfigurationAdministators group by default

BRKVIR-2700 74

Use the local SSO account during the HX installation

aldanch@cnmi.com

• Local SSO account has root-level privileges to vCenter

• It’s a member of the SystemConfigurationAdministators group by default

BRKVIR-2700 75

Considerations when purchasing HX with vSphere licenses

HX240cM4

1 6 12 18 24

HyperFlex

ESXi 6.0 U1b , ESXi 6.0 U2 P3

BRKVIR-2700 76

HX240cM4

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Enterprise Plus factory-installed on HX nodes

No physical PAK or license keys provided

BRKVIR-2700 77

HX240cM4

1 6 12 18 24

HyperFlex

vSphere OEM license keys are obfuscated

i.e. 0N085-XXXXX-XXXXX-XXXXX-10LHH

All factory-installed HX nodes share the SAME OEM license key

BRKVIR-2700 78

HX240cM4

1 6 12 18 24

HyperFlex

No pre-installation of ESXi Standard, Essentials Plus, or ROBO editions

vSphere OEM license keys are obfuscated

i.e. 0N085-XXXXX-XXXXX-XXXXX-10LHH

All factory-installed HX nodes share the SAME OEM license key

BRKVIR-2700 79

Considerations when purchasing HX without vSphere licenses

HX240cM4

1 6 12 18 24

HyperFlex

BRKVIR-2700 80

HX240cM4

1 6 12 18 24

HyperFlex

ESXi license keys must be supplied by end-customer

BRKVIR-2700 81

HX240cM4

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Foundation factory-installed on HX nodes

BRKVIR-2700 82

HX240cM4

1 6 12 18 24

HyperFlex

OEM version of ESXi 6.0 Foundation factory-installed on HX nodes

After initial setup, license can be changed to any supported edition

BRKVIR-2700 83

Set ESXi to use an external syslog server

Confirm on your syslog server that logs are being received

BRKVIR-2700 84

Configuring an external syslog server can also be done using the vSphere client

BRKVIR-2700 85

Reset the ESXi Host root password from the default on all HX nodes

BRKVIR-2700 86

Alternatively, reset the ESXi Host root password from the DCUI

BRKVIR-2700 87

Alternatively, reset the ESXi Host root password from the DCUI

BRKVIR-2700 88

If the ESXi version needs to be upgraded, use the bundle.zip file available at Cisco.com

BRKVIR-2700 89

Set reserve vSphere HA failover CPU and Memory capacity

• n/cluster-size

• Ex. For a 4 node cluster with N+1, reserve ¼ or 25%

• Ex. For an 8 node cluster with N+1, reserve 1/8 or 13%

• Try to avoid using “host failures” admission control as it becomes inflexible in many cases with large reservations and wastes resources with unbalanced clusters

BRKVIR-2700 90

Accept the default setting Datastore for Heartbeating (automatic)

BRKVIR-2700 91

Leave default DRS Settings as is – Fully automated

BRKVIR-2700 92

Enable EVC BEFORE adding any workloads to the HX cluster

• Prepares the cluster for

• Adding HX nodes with newer CPUs (v4 CPU, Broadwell)

• Upgrading CPUs on existing HX nodes

• Use lowest CPU model in cluster (currently v3 CPU, Haswell)

BRKVIR-2700 93

HX installer will create vSwitches, port groups and assign VLANs and vmnics accordingly

BRKVIR-2700 94

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-ACISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-B

HX240cM4

1 6 12 18 24

HyperFlex

Active Stby Active Active Stby Active Active Stby

vmnic0 vmnic1vmnic2 vmnic3vmnic4 vmnic6 vmnic5 vmnic7

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 95

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-B

HX240cM4

1 6 12 18 24

HyperFlex

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 96

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-B

HX240cM4

1 6 12 18 24

HyperFlex

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 97

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-B

HX240cM4

1 6 12 18 24

HyperFlex

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 98

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

ID FI-B

HX240cM4

1 6 12 18 24

HyperFlex

hv-mgmt-a

storage-data-a

hv-mgmt-b

storage-data-b

vm-network-b

vm-network-a

hv-vmotion-a

hv-vmotion-b

Port 1 Port 2

VIC 1227

BRKVIR-2700 99

Policy Map

Uplink Interface

QoSSystem Class

Enable jumbo frames end-to-end to support HX storage and vMotion traffic

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

HX240cM4

1 6 12 18 24

HyperFlex

ESXi Hypervisor

BRKVIR-2700 100

vMotion and VM networks are supported to migrate to a distributed switch (vDS or Nexus 1000v)

Mirror the configuration from the standard vSwitch settings

BRKVIR-2700 101

For vMotion, set the MTU size to 9000 at the dvSwitch and VMkernel port group levels

BRKVIR-2700 102

• Prerequisites

• UCS

• vSphere

Agenda

HX Data Platform

After HX deployment, SSH into the HX installer and execute the post_install script

root@Cisco-HX-Data-Platform-

Installer:~# post_install

BRKVIR-2700 105

After HX deployment, SSH into the HX installer and execute the post_install script

root@Cisco-HX-Data-Platform-

Installer:~# post_install

• Helps streamline some of the post-

install tasks

• Can be re-run to add additional

VLANs on existing HX nodes

• Additional converged or compute

nodes would require manual

configuration

BRKVIR-2700 106

Storage

Cluster

Management IP

Configure ASUP to get information about failures and cluster health

SCVMYes

asup.hyperflex.io

saipan-hx-cluster@cnmi.com

BRKVIR-2700 107

For ASUP, enter the SMTP information, enable ASUP services, add email recipients, and review

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 108

root@SpringpathController:~# stcli services smtp set --smtp

asup.hyperflex.io --fromaddress saipan-hx-cluster@cnmi.com

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 109

root@SpringpathController:~# stcli services smtp set --smtp

asup.hyperflex.io --fromaddress saipan-hx-cluster@cnmi.com

root@SpringpathController:~# stcli services smtp show

smtpServer: asup.hyperflex.io

fromAddress: saipan-hx-cluster@cnmi.com

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 110

root@SpringpathController:~# stcli services asup enable

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 111

root@SpringpathController:~# stcli services asup recipients set –-recipients

saipan-it@cnmi.com

root@SpringpathController:~# stcli services asup recipients add –-recipients

aldanch@cnmi.com

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 112

root@SpringpathController:~# stcli services asup recipients set –-recipients

saipan-it@cnmi.com

root@SpringpathController:~# stcli services asup recipients add –-recipients

aldanch@cnmi.com

root@SpringpathController:~# stcli services asup show

recipientList:

----------------------------------------

saipan-it@cnmi.com

----------------------------------------

aldanch@cnmi.com

----------------------------------------

enabled: True

saipan-hx-cluster

asup.hyperflex.io

saipan-it@cnmi.com

aldanch@cnmi.com

BRKVIR-2700 113

Send a test email to verify that SCVM can send emails to the test recipient

root@SpringpathController:~# echo "i am testing email

capability of this SCVM" | mailx -s "this is a test"

aldanch@cnmi.com

BRKVIR-2700 114

aldanch@cnmi.com

• Check the target mailbox

BRKVIR-2700 115

aldanch@cnmi.com

• Check the target mailbox

root@SpringpathController:~# sendasup –t

• You can search ASUP in a text editor using CTRL+F

BRKVIR-2700 116

Change the SCVM password from the default during post deployment

Running this command will apply the password to all SCVMs

BRKVIR-2700 117

Manage your HX datastores through the HX plugin or HX Dashboard – i.e, creation, resizing, renaming

BRKVIR-2700 118

Manage your HX datastores through the HX plugin or HX Dashboard – i.e, creation, resizing, renaming

BRKVIR-2700 119

Monitor free space on the HX cluster and put capacity planning measures in place

BRKVIR-2700 120

Use HX Data Platform to create VM snapshots

BRKVIR-2700 121

BRKVIR-2700 122

BRKVIR-2700 123

When cloning VMs, use HX ReadyClones

BRKVIR-2700 124

When cloning VMs, use HX ReadyClones

BRKVIR-2700 125

When cloning VMs, use HX ReadyClones or normal VMware Clone Wizard

BRKVIR-2700 126

• Prerequisites

• UCS

• vSphere

Agenda

Troubleshooting Scenarios

Confirm network connectivity if problems are encountered during install

• SSH to HX installer

• Ping the UCSM cluster IP

• Ping the vCenter server IP

• Ping all ESXi hosts management IP (vmk0)

• Verify DNS resolution functional:

• vCenter

• SSO URI

• UCSM

• ESXi hosts

BRKVIR-2700 129

Synchronize time fails if not using an internal time source

• Use a reliable NTP time source (internal server or domain controller)

• Avoid using public NTP servers

BRKVIR-2700 130

VM migration fails due to missing VMkernel adapter

• vMotion VMkernel adapter is not created automatically unless post_install script is run after deployment or if it is manually created

BRKVIR-2700 131

Create the VMkernel adapter either in CLI or through the vSphere client with MTU 9000

[root@esxi-01:~] esxcfg-vmknic -a -i 192.168.4.158 -n

255.255.255.0 –m 9000 -p vMotion

BRKVIR-2700 132

Confirm that the vMotion VLAN is on the port-group, vNICs, uplink interfaces, and switchports

BRKVIR-2700 133

Confirm that the vMotion VLAN is on the port-group, vNICs, uplink interfaces, and switchports

BRKVIR-2700 134

Confirm that the vMotion VLAN is on the port-group, vNICs, network uplink ports, and switchports

BRKVIR-2700 135

BRKVIR-2700 136

BRKVIR-2700 137

Test vMotion connectivity with vmkping between HX nodes

A regular vmkping will confirm initial connectivity between HX nodes

BRKVIR-2700 138

Test vMotion connectivity with vmkping between HX nodes

A jumbo vmkping will confirm jumbo frames between HX nodes is functional

BRKVIR-2700 139

If it fails with jumbo MTU, check vSwitch, vNICs, QoS, northbound switch MTU and CoS for misconfig

BRKVIR-2700 140

BRKVIR-2700 141

BRKVIR-2700 142

BRKVIR-2700 143

vMotion traffic is localized to FI-A unless a network failure event causes a failover to FI-B

BRKVIR-2700 144

Other issues that would cause migration to fail

VM Level Issues

• Incompatible CPUs – mixed CPU families with HX or compute-only

• VM port group is not created on destination host

• Additional VMDKs (external storage) not available to all hosts

• VM connected to physical hardware (PCI passthrough, USB, etc)

• Not enough resources to accommodate VM

BRKVIR-2700 145

If the HX node is NOT re-imaged with the HX custom ESXi ISO, the ESXi configuration step fails

BRKVIR-2700 146

• Check the latest Cisco documentation

BRKVIR-2700 147

• Use the HX custom ISO file as it contains an integrated install script that follows the factory imaging procedures

BRKVIR-2700 148

• Mount this ISO via KVM and re-image all HX nodes (non-interactive)

BRKVIR-2700 149

• Mount this ISO via KVM and re-image all HX nodes (non-interactive)

• WARNING: Re-imaging ESXi removes the OEM license key

• Open a TAC case to recover the license

BRKVIR-2700 150

During HX upgrade, java.net.ConnectionException: Connection refused error message appears

BRKVIR-2700 151

• Caused by SSH on ESXi being disabled

BRKVIR-2700 152

• Caused by SSH on ESXi being disabled

• Enable SSH on all HX nodes to continue with the HX upgrade

BRKVIR-2700 153

HX Upgrades: Multiple Clusters Registered to One vCenter

• Each cluster must be independently bootstrapped

• First bootstrap will update the plugin for all clusters, but you must ensure each cluster has the latest packages installed by running cluster-bootstrap.sh

BRKVIR-2700 154

HX Upgrades: UCS Not Running Expected Version

This will occur if you fail to upgrade the UCS infrastructure before beginning the upgrade. Follow the earlier steps to upgrade the UCS infrastructure first.

BRKVIR-2700 155

HX Upgrades: Node Maintenance Mode Failed

Will occur after a 1 hour timeout if you fail to vacate the running VMs from that host.

Re-try the upgrade again and be sure to check on the status of VM migration referencing vCenter and the event log for the current node

• If DRS is enabled, a VM might get stuck due to incompatibility to vMotion (network problem, ISO mounted, CPU features missing, etc.)

• If DRS is disabled or enabled but not in fully automatic mode, you will have to manually migrate your VMs on each node

BRKVIR-2700 156

HX Upgrades: Error Occurred While Configuring UCS

Occurs after a 2 hour timeout if the server fails to complete firmware upgrades & full boot into ESXi

1. Check UCSM for the progress of the firmware update

2. Remediate the issue

3. Wait for ESXi to boot

4. Exit maintenance mode

5. Re-try upgrade again

BRKVIR-2700 157

HX Upgrades: Validation Failed – UCS Version

• Ensure you have uploaded the packages to UCSM

• Check the exact syntax of your input into target version textbox

• E.g. 3.1(2b)

BRKVIR-2700 158

HX Upgrades: No Connection Error

1. If a converged node is in maintenance mode with controller VM shutdown

2. Didn’t upgrade HXDP first before FW in split upgrade

3. Any stCtlVM not running stNodeMgr• Can run “status stNodeMgr” to see if it is

running

• If it isn’t running, it is preferred to restart the CVM

• Do not attempt to do this without TAC assistance!

• You might bring down an online cluster if you fail to check failure tolerances

BRKVIR-2700 159

HX Upgrades: Monitoring Progress

• Best to watch progress in vCenter• If your session is disconnected, you can reconnect to the plugin and resume monitoring the

upgrade

• You can also monitor via the CLI• CLI session will disconnect when the controller VM you are connected to is rebooted

• stcli cluster upgrade --status

• Based on current step, you can find more details within that management console (UCSM, ESX event log, etc.)

• Don’t assume the upgrade is frozen – many times we are healing behind the scenes• Use “stcli cluster storage-summary --detail” to check healing status

BRKVIR-2700 160

• HyperFlex Field Install Guide http://cisco.box.com/v/hx-install

• HyperFlex Field Upgrade Guidehttp://cisco.box.com/v/hx-upgrade

• HyperFlex Customer Cleanup/Reinstall Guidehttp://cisco.box.com/v/hx-cleanup

HyperFlex Field Guides

• Prerequisites

• UCS

• vSphere

Agenda

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

BRKVIR-2700 163

Continue Your Education

• Demos in the Cisco campus

• Visit the Hub (2.2) for:

• DevNet Zone related sessions and demos

• Walk-in Self-Paced Labs

• Technical Solutions Clinic

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

• Join the session’s Spark room

BRKVIR-2700 164

and Cisco

BRKVIR-2700 165

Cisco SparkAsk Questions, Get Answers, Continue the Experience

Use Cisco Spark to communicate with the Speaker and fellow participants after the session

Download the Cisco Spark app from iTunes or Google Play

1. Go to the Cisco Events mobile app

2. Find this session

3. Click the Spark button under Speakers in the session description

4. Enter the room, room name = BRKVIR-2700

5. Join the conversation!

The Spark Room will be open for 2 weeks after Cisco Live

BRKVIR-2700 166

Thank You

Appendix

Links• Documentation Roadmap:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HX_Documentation_Roadmap/HX_Series_Doc_Roadmap.html

• Pre-install Checklist: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_preinstall_checklist/b_HyperFlexSystems_pre_install_checklist_1_8.html

• Getting Started Guide: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/GettingStartedGuide/1-8/b_HyperFlexSystems_GettingStartedGuide_1_8_c.html

• Configuration Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-and-configuration-guides-list.html

• End-User Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-user-guide-list.html

• Cisco Validated Design for HCI: http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-cloud-computing/hyperconverged-infrastructure-design.html

• HX220/HX240 Node Installation Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-series/products-installation-guides-list.html

BRKVIR-2700 170

Links• Release Notes: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-

release-notes-list.html

• Upgrade Guides: http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-installation-guides-list.html

• Disjoint Layer 2: http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/white_paper_c11-692008.html

• Cisco HyperFlex HX-Series Hardware and Software Interoperabiliy Matrix in UCSM Release 3.1(2): http://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/interoperability/matrix/b_hcl_HX-rel-3-1-2.pdf

• Video: Cisco HyperFlex Day Zero Configuration: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_day_zero_configuration.html

• Video: Cisco HyperFlex Compute Only Node Addition: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/videos/cisco_hyperflex_compute_only_node_addition.html

BRKVIR-2700 171

Appendix: Prerequisites

HyperFlex Pre-Install Checklist: UCS Fabric Interconnect Information

Uplink Switch Model

Fabric Interconnect Cluster IP Address

UCS FI-A Address

UCS FI-B Address

Pool for KVM IPs (One per HX node is required)

Subnet Mask

Default Gateway

MAC Pool Prefix

User ID

Password

BRKVIR-2700 173

HyperFlex Pre-Install Checklist: VLAN Information

Network VLAN ID Description

hx-inband-mgmtUsed for management traffic between ESXi, HX, and

vCenter. Must be routable

hx-storage-data Used for storage traffic and requires L2

hx-vmotion Used for live migration of VMs

vm-network Used by VM/Application network

additional network

BRKVIR-2700 174

HyperFlex Pre-Install Checklist: IP Address Information

Management Network Storage Data Network

ESXi vmk0 Storage Controller Eth0 ESXi vmk1 Storage Controller Eth1

Node 1 IP Address

Node 2 IP Address

Node 3 IP Address

Node 4 IP Address

Subnet Mask Subnet Mask

Default Gateway Default Gateway

BRKVIR-2700 175

HyperFlex Pre-Install Checklist: IP Address Information

• 3 VLANs recommended – 2 required

• Number of IP addresses• VLAN X (management): 2 x number of nodes + 2

• E.g. 4 node cluster needs 10 IPs. 8 node cluster needs 18 IPs

• VLAN Y (storage data): 2 x number of nodes + 1 [can all be non-routable/private]• E.g. 4 node cluster needs 9 IPs. 8 node cluster needs 17 Ips

• VLAN Z (vMotion): 1 x number of nodes [can be non-routable/private]

• UCSM VLAN (typically X): 1 x number of nodes + 3 [for new deployments only]

• In total:• E.g. 4 node cluster needs 30 IPs, 8 node cluster needs 54 IPs

• Be very careful IPs fall in proper subnet and check for duplicate IPs!

BRKVIR-2700 176

HyperFlex Pre-Install Checklist: vMotion Network IP Address (configured by post_install script)

IP Address HX Node

IP Address Server 1

IP Address Server 2

IP Address Server 3

IP Address Server 4

Subnet Mask: Default Gateway:

If vMotion network is on its own dedicated TCP/IP stack, include the subnet mask and default gateway here

BRKVIR-2700 177

HyperFlex Pre-Install Checklist: Storage Cluster Information

Storage Cluster

Storage Cluster Name

Storage Cluster Management IP Address

Storage Cluster Data IP Address

BRKVIR-2700 178

HyperFlex Pre-Install Checklist: ESXi Information

Hypervisor Credentials Default

Username root

Password Cisco123

Hypervisor Credentials New

Username root

Password

BRKVIR-2700 179

HyperFlex Pre-Install Checklist: vCenter Information

vCenter Configuration

vCenter Server

https://<IP address>

(default ports are required)

vCenter admin user name

username@domain

vCenter Datacenter Name

Datacenter in vCenter where HX cluster will reside

Cluster Name

ESXi cluster name under the Datacenter where that the

HX nodes will be added to

BRKVIR-2700 180

HyperFlex Pre-Install Checklist: SSO Information

Single Sign On (SSO)

SSO Server URL

• Only required if SSO URL is not reachable

• Automatic for ESXi 6.0 and later

BRKVIR-2700 181

HyperFlex Pre-Install Checklist: Network Information

Network Services

DNS Server(s)

IP Address

NTP Server(s)

IP Address

Time Zone

Example: US/Eastern, US/Pacific

BRKVIR-2700 182

HyperFlex Pre-Install Checklist: Auto Support Information

Auto Support

Enable Auto Support

Yes/No

SMTP Server

Example: mailhost.company.com

Mail Sender

Example: sender@company.com

BRKVIR-2700 183

Physical Requirements

• 2-4 uplink connections for FIs

• Each FI will need at least two 10 Gb connection to existing network

• Each HX node will need at least two 10 Gb connections

• Optical or Twinax cables are supported

• Do NOT mix optical and twinax cables

• Example: • HX node connects to FI-A with twinax and FI-B with optical

• FI A connects to Switch 1 with twinax and Switch 2 with optical

BRKVIR-2700 184

Physical Requirements: route network cables

• Connect server port 1 to FI-A (right port)

• Connect server port 2 to FI-B (left port)

• Use the same port numbers for each node

• Port 1 on FI-A and FI-B to server 1

• Port 2 on FI-A and FI-B to server 2

• …

• Connect uplink ports on each FI

• Suggest to use port 17 or higher

• 10Gb uplinks are strongly advised

BRKVIR-2700 185

Physical Requirements: Rack Space

• For HX220c Cluster

• 2RU for ToR switch pair

• 2RU for 6248 pair

• 4RU for 6296 pair

• 1RU for each HX220c node

• For HX240c Cluster

• 2RU for ToR switch pair

• 2RU for 6248 pair, 4RU for 6296 paisr

• 2RU for each HX240c node

• 4RU for 5108 chassis for up to eight B200 M4 ½ width blades

• 2 C13/C14 power connections to 15 AMP circuit per device

BRKVIR-2700 186

Verify datacenter space for the HX cluster (rack space, power, cooling, networking cables)

• UCS FI 6248UP = 1 RU

• UCS FI 6296UP = 2 RU

• HX220c / C220 M4 = 1 RU

• HX240c / C240 M4 = 2 RU

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

CISCO UCS 6296UP

7 8HX220c

Console!

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

600W to

Max power per component

BRKVIR-2700 187

Verify datacenter space for the HX cluster (rack space, power, cooling)

• 5108 Chassis = 4 RU

• B200 M4 = ½ width 1 RU

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

! ResetConsole

UCS B200 M4

Max power output per chassis

BRKVIR-2700 188

Appendix: UCS

For new installs, download the UCS A-, B-, and C-firmware bundles and stage them in UCS Manager

BRKVIR-2700 190

For new installs, download the UCS A-, B-, and C-firmware bundles and stage them in UCS Manager

BRKVIR-2700 191

EXAMPLE – Use Auto-Install to upgrade the Infrastructure firmware – Only for new installs

BRKVIR-2700 192

EXAMPLE – Use Auto-Install to upgrade Server firmware for converged and compute nodes

BRKVIR-2700 193

Confirm the desired package version has been fully applied to all UCS components

BRKVIR-2700 194

HX installer configures QoS Policies that will be mapped to vNIC templates

BRKVIR-2700 195

HX installer also creates MAC pools with 100 MAC addresses for each vNIC template

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

HX240cM4

1 6 12 18 24

HyperFlex

BRKVIR-2700 196

To add new vNICs, run and complete the UCSM and ESXi configuration only in the HX installer

BRKVIR-2700 197

To add new vNICs, run and complete the UCSM and ESXi configuration only in the HX installer

BRKVIR-2700 198

After UCS and ESXi configurations have completed, create new MAC pools for each new vNIC

BRKVIR-2700 199

Create additional vNIC templates under the HX cluster sub-org

BRKVIR-2700 200

Create a pair of vNIC templates with 1:1 mapping to FI-A or FI-B

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

BRKVIR-2700 201

Add newly created vNIC templates to HyperFlexLAN connectivity policy

BRKVIR-2700 202

Name the vNIC, map the correct vNIC template, and select the HyperFlex adapter policy

BRKVIR-2700 203

Accept the changes to place the HX nodes into Pending Activities to prepare for staged reboots

Select the checkboxes for all the affected servers and click OK to apply the changes

BRKVIR-2700 204

Once the vNICs/vHBAs have been created, go back into HX installer and deploy HX

BRKVIR-2700 205

Once the vNICs/vHBAs have been created, go back into HX installer and deploy HX

BRKVIR-2700 206

If HX cluster is already deployed, verify that the HX cluster is healthy prior to rebooting the HX node

BRKVIR-2700 207

In vSphere web client, place the HX Node into HX Maintenance Mode

• If DRS is enabled, VMs will be migrated automatically and the stctlvm will be shut down gracefully

• If there is no DRS, manually migrate all the VMs off beforehand (except stctlvm)

• Confirm HX node has successfully entered maintenance mode

BRKVIR-2700 208

At UCSM, go to Pending Activities, select the server, click the checkbox for Reboot Now, and click OK

BRKVIR-2700 209

Once the HX node is back online, update its PCI Devices by adding the LSI3108_1 device

BRKVIR-2700 210

Reboot the HX node in order for the changes to take affect

BRKVIR-2700 211

Once the HX node is back online, exit HX maintenance mode

NOTE: If HX maintenance mode does not work, then simply exit maintenance mode the old fashioned way

BRKVIR-2700 212

Select the HX node’s stctlvm and edit its hardware settings to remove and re-add its PCI card

BRKVIR-2700 213

Select the HX node’s stctlvm and edit its hardware settings to remove and re-add its PCI card

BRKVIR-2700 214

Power on the stctlvm and monitor the HX storage cluster until it displays “healthy”

Once cluster is healthy, it is safe to move on to next HX node

BRKVIR-2700 215

Appendix: vSphere

vSwitch Settings vSwitch

NameVMNIC

Active

Standby

PolicyCoS

Packet

vswitch-hx-inband-mgmt vmnic0 Active hv-mgmt-a 1500 Silver 2 Yes

vswitch-hx-inband-mgmt vmnic1 Standby hv-mgmt-b 1500 Silver 2 Yes

vswitch-hx-storage-data vmnic2 Standby storage-data-a 9000 Platinum 5 No

vswitch-hx-storage-data vmnic3 Active storage-data-b 9000 Platinum 5 No

vswitch-hx-vm-network vmnic4 Active vm-network-a 1500 Gold 4 Yes

vswitch-hx-vm-network vmnic5 Active vm-network-b 1500 Gold 4 Yes

vmotion vmnic6 Active hv-vmotion-a 9000 Bronze 1 Yes

vmotion vmnic7 Standby hv-vmotion-b 9000 Bronze 1 Yes

BRKVIR-2700 217

Under vswitch-hx-vm-network, create additional port groups for other VM traffic for each HX node

BRKVIR-2700 218

Ensure VLANs are created and allowed in the northbound switches and in UCS ManagerInterface port-channel 201

switchport trunk allowed vlan 4-6, 10, 20, 67, 500, 1000, 2000, 3000

Interface port-channel 202

switchport trunk allowed vlan 4-6, 10, 20, 67, 500, 1000, 2000, 3000

BRKVIR-2700 219

For Disjoint L2 VMs, create a custom vSwitch in CLI, then add vmnics, and create port groupsSSH into the HX node and create a vSwitch with a custom name[root@esxi-01:~] esxcli network vswitch standard add -v vswitch-dmz-network

Repeat steps on every HX nodeBRKVIR-2700 220

ESXi Upgrade: Copy the bundle.zip file to a datastore that the HX node can access

FTP, SFTP,

TFTP, SCP

vSphere Client

HX Datastore

HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

BRKVIR-2700 221

Enter the HX node into HX Maintenance Mode to prepare for the upgrade

BRKVIR-2700 222

SSH into the HX node, review the bundle, and then perform the upgrade[root@hx-esxi:~] esxcli software sources profile list -d /vmfs/volumes/hx-datastore/HX-

Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip

Name Vendor Acceptance Level

------------------------------------------------------------------------

Vmware-ESXi-60U2-4192238-Cisco-Custom-HX Cisco PartnerSupported

[root@hx-esxi:~] esxcli software profile update -d /vmfs/volumes/hx-datastore/HX-Vmware-

ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip -p Vmware-ESXi-60U2-4192238-Cisco-

Custom-HX

Update Result Message: The update completed successfully, but the system needs to be

rebooted for the changes to be effective. Reboot Required

[root@hx-esxi:~] reboot

BRKVIR-2700 223

Once the HX node is back online, verify the ESXi version and then exit HX maintenance mode

BRKVIR-2700 224

Once the HX node is back online, verify the ESXi version and then exit HX maintenance mode

BRKVIR-2700 225

Check the health of the HX cluster

Repeat this process on each HX node until all are upgraded to the same ESXi version

BRKVIR-2700 226

Appendix: HX Data Platform

Sample output of post_install scriptroot@Cisco-HX-Data-Platform-Installer:~# post_install

Logging in to controller 10.255.255.253

HX CVM root Password for:

Getting ESX hosts from HX cluster...

vCenter URL: 10.255.255.254

Enter vCenter username (user@domain):

administrator@vsphere.local

vCenter Password:

Found datacenter CNMI

Found cluster Saipan-HX-Cluster

Enable HA/DRS on cluster? (y/n) y

Disable SSH warning? (y/n) y

Configure ESXi logging onto HX datastore? (y/n) y

No datastores found

Creating datastore...

Name of datastore: Saipan-DS-1

Size (GB): 1024

Storing logs on datastore Saipan-DS-1

Creating folder [Saipan-DS-1]/esxi_logs

Add vmotion interfaces? (y/n) y

Netmask for vMotion: 255.255.255.0

VLAN ID: (0-4096) 4000

vMotion IP for 10.255.255.250: 192.168.255.250

Adding vmotion to 10.255.255.250

Adding vmkernel to 10.255.255.250

vMotion IP for 10.255.255.251: 192.168.255.250

vMotion IP for 10.255.255.252: 192.168.255.252

Add VM network VLANs? (y/n) y

Attempting to find UCSM IP

Found UCSM 10.255.255.200, logging with username admin. Org is

Saipan-HX-Cluster

UCSM Password:

Port Group Name to add: Saipan-Prod

VLAN ID: (0-4096) 100

Adding VLAN 100 to FI

Adding VLAN 100 to vm-network-a VNIC template

Adding VLAN 100 to vm-network-b VNIC template

Adding Saipan-Prod to 10.255.255.250

Add additional VM network VLANs? (y/n) n

BRKVIR-2700 228

Sample output of post_install script (cont’d)Enable NTP on ESX hosts? (y/n) y

Starting ntpd service on 10.0.0.251

Send test email? (y/n) y

Recipient email address: aldanch@cnmi.local

Sending test email to aldanch@cnmi.local

Verify if email recieved.

Validating cluster health and configuration...

Checking MTU settings

ESX password:

Pinging 192.168.100.251 from vmk1

Setting vnic2 to active and vmnic3 to standby

Pinging 192.168.100.253 with mtu 8972 from vmk1

Setting vmnic3 to active and vnic2 to standby

BRKVIR-2700 229

Sample output of post_install script (cont’d)Network Summary:

Host: 10.0.0.251

vswitch: vswitch-hx-inband-mgmt - mtu: 1500 - policy:

loadbalance_srcid

vmnic0 - 1 - CNMI-FI-A - active

vmnic1 - 1 - CNMI-FI-B - standby

Portgroup Name - VLAN

Storage Controller Management Network - 10

Management Network - 10

vswitch: vswitch-hx-vm-network - mtu: 1500 - policy:

loadbalance_srcid

vmnic5 - 1 - CNMI-FI-B - active

Saipan-Prod-10 - 10

Saipan-Dev-20 - 20

vswitch: vmotion - mtu: 9000 - policy: loadbalance_srcid

vmotion - 101

vswitch: vswitch-hx-storage-data - mtu: 9000 - policy:

loadbalance_srcid

vmnic2 - 1 - CNMI-FI-A - standby

Storage Controller Data Network - 100

Storage Hypervisor Data Network - 100

Host: 10.0.0.252

loadbalance_srcid

Saipan-Prod-10 - 10

Saipan-Dev-20 - 20

vmotion - 101

loadbalance_srcid

BRKVIR-2700 230

Sample output of post_install script (cont’d)Host: 10.0.0.253

loadbalance_srcid

Saipan-Prod-10 - 10

Saipan-Dev-20 - 20

vmotion - 101

loadbalance_srcid

Host: 10.0.0.251

No errors found

Host: 10.0.0.252

No errors found

Host: 10.0.0.253

No errors found

Controller VM Clocks:

stCtlVM-FCH2017V1AB - 2016-12-25 12:25:10

stCtlVM-FCH2017V2CD - 2016-12-25 12:25:20

stCtlVM-FCH2017V3EF - 2016-12-25 12:25:30

Cluster:

Version - 1.8.1c-19725

Model - HX220C-M4S

Health - HEALTHY

Access Policy - LENIENT

ASUP enabled - True

SMTP Server - mail.cnmi.local

BRKVIR-2700 231

Appendix: Troubleshooting Scenarios

HX Upgrades: Configure Lenient Mode

Lenient mode is now default in 1.8(x)

1. SSH to any one of the controller VMs and login as root

2. Check if lenient mode is already configured

~/#stcli cluster get-cluster-access-policy

3. If set to strict, change to lenient. If already lenient, no further action is required

~/#stcli cluster set-cluster-access-policy --name lenient

4. Confirm the change by typing “stcli cluster info | grep -i policy”

BRKVIR-2700 233

HX Upgrades: Upgrading from 2.2(6f)

• Upgrade time will increase when upgrading from 2.2(6f) due to deep discovery of storage controller and VIC

• Add up to 60 additional minutes per node to your planning

• You might see node reboot multiple times while installing firmware

• The process is completely automated but can be monitored from FSM tab

• This must only be done once in a cluster’s lifetime and only if starting at 2.2(6f)

BRKVIR-2700 234

Create Validation fails due to DRIVER-Check mpt3sas and DRIVER-Check megaraid_sas

BRKVIR-2700 235

• Driver list in SCVM not updated

BRKVIR-2700 236

• Reboot the SCVMs to properly load the driver list

• Log into vSphere client and gracefully reboot all SCVMs

BRKVIR-2700 237

• Reboot the SCVMs to properly load the driver list

• Log into vSphere client and gracefully reboot all SCVMs

• Once all SCVMs are up and running, click “Retry Create Validation” to continue the HX installation

BRKVIR-2700 238

Deploy fails due to ‘disk /dev/… is formatted as a storfs disk’

BRKVIR-2700 239

• Occurs when ESXi is re-installed on the HX node

BRKVIR-2700 240

• Click Edit Configuration Continue

BRKVIR-2700 241

• Check the box “Clean up disk partitions” in the Advanced Configuration

BRKVIR-2700 242

• Check the box “Clean up disk partitions” in the Advanced Configuration

Click Continue Click “Retry

Deploy” at the top of the page

BRKVIR-2700 243

HyperFlex Configuration Best Practices and...

Documents

Transcript of HyperFlex Configuration Best Practices and...

Cisco HyperFlex Systems · © 2016 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 31 White Paper Cisco HyperFlex Systems Cisco HyperFlex

Smart play hyperflex-program-guide-v2.5

Implementing SMB File Services On Cisco Hyperflex Using ... · VM configuration best practices The following section describes best-practices considerations for configuring the VM:

Cisco HyperFlex Hyperconverged System for Virtual Server ...

Living on the Cisco HyperFlex Edge · VMware license Hyperconverged infrastructure automation Integration with existing network Fixed configuration ROBO: Cisco HyperFlex HX220c Computing:

Hypervisors networking: best practices for interconnecting with Cisco switches BRKVIR-2019 Mioljub Jovanovic – Technical Leader.

Cisco Hyperflex HX240c M5 Node Spec Sheetcontent.etilize.com/Manufacturer-Brochure/1040985540.pdf · 2017-11-07 · Cisco HyperFlex HX240c M5 Node OVERVIEW 3 OVERVIEW Cisco HyperFlex™

BRKVIR-3013 Deploying and Troubleshooting the Nexus 1000v Virtual Switch (VSphere and HyperV)

Capacity Management in Cisco HyperFlex White Paper

Cisco HyperFlex Systems HX-Series Overview

Hyperflex Wetsuits

Scalar Cisco Hyperflex Presentation, May 13 2016, Part III: Scalar Lunch & Learn Seminar Series: HyperConverged Systems: Cisco HyperFlex Systems

CISCO HYPERFLEX AND THALES eSECURITY ...go.thalesesecurity.com/rs/480-LWA-970/images/Cisco... · The Cisco HyperFlex system delivers hyperconvergence, ...

Cisco HyperFlex Integration with Citrix Cloud and Citrix ...

Cisco Hyperflex HX220c M5 Spec Sheet · The Cisco HyperFlex HX220c M5 Node is shown in Figure 1. The HX220c M5 servers extend the capabilities of Cisco’s HyperFlex portfolio in

Cisco HyperFlex All-Flash Systems for Oracle Real ... · PDF fileOracle Database is the choice ... This reference architecture provides a configuration that ... protection techniques

Cisco Hyperflex - AmCham El Salvadoramchamsal.com/files/presentations/15ce6066ba8bc6.pdf · Cisco HyperFlex Rapid Pace in Leadership Gartner, Magic Quadrant for Hyperconverged Infrastructure,

Cisco Hyperflex HX240c M4 Node Spec Sheet · 2017-09-08 · Cisco HyperFlex HX240c M4 Node OVERVIEW 3 OVERVIEW Cisco HyperFlex™ Systems unlock the full potential of hyperconvergence.

TechWiseTV Workshop: Cisco HyperFlex Systems

Cisco HyperFlex for Citrix Cloud services Cloud... · Cisco HyperFlex for Citrix Cloud services Cisco Intersight HyperFlex Connect 2 x Citrix Cloud connectors Active Directory Virtual