White Paper

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 17

Virtualized SAP HANA on FlashStack Converged Infrastructure

Technical Considerations and Best Practices

August 2017

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 17

Contents Introduction

Purpose of this document Audience Document reference Test environment

Reference architecture Storage layer Server and network layers Virtual machine, OS and application layers

Sizing Compute Storage

For more information Certified SAP HANA Hardware Directory SAP HANA TDI documentation SAP Notes Other links

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 17

Introduction

Support for running SAP HANA virtualized on VMware vSphere has been available since the release of SAP HANA

Support Package Stack (SPS) 05. Support has been available starting with VMware vSphere 5.1 for nonproduction

scenarios. Most recently, support has been available with SAP HANA SPS 12 or later for production single–virtual

machine and multiple–virtual machine use cases with up to 4 terabytes (TB) of main memory. Customers now can

gain the benefits of virtualization for their SAP HANA environments and also achieve cost optimization through

more efficient resource utilization.

Both SAP HANA appliance and SAP HANA Tailored Datacenter Integration (TDI) delivery methods are supported

for SAP HANA deployments on VMware vSphere. Virtualized SAP HANA systems, whether delivered preinstalled

and configured by a certified vendor or implemented by the customer using existing hardware listed in the Certified

and Supported SAP HANA Hardware Directory, should be verified with the SAP HANA Hardware Configuration

Check Tool (HWCCT) to qualify for support from SAP. The main use of SAP HANA on VMware is within the TDI

framework. Increasing numbers of customers are taking advantage of the extended flexibility that comes with

running SAP HANA TDI.

IDC confirmed in a recent survey (IDC US42198216) that Cisco is the leading vendor for SAP HANA infrastructure.

In line with the current trend toward converged infrastructure to further simplify and support multitenant workload

deployments, Cisco and Pure Storage have partnered to deliver FlashStack converged infrastructure as a platform

for a variety of SAP workloads, including fully virtualized workloads. FlashStack uses best-in-class server and

network components integrated with the programmability of the Cisco Unified Computing System™

(Cisco UCS®)

backed by high-performance all-flash storage from Pure Storage. With Cisco UCS servers and Pure Storage

FlashArray//m series listed in the Certified and Supported SAP HANA Hardware Directory’s Certified Appliances

and Certified Enterprise Storage sections respectively, FlashStack provides an optimized platform for

implementations of SAP HANA TDI.

Purpose of this document

This document provides configuration guidelines for implementing virtualized SAP HANA on FlashStack converged

infrastructure. It describes the best practices to be followed for the storage, server, network, virtualization, OS, and

application layers for an optimal configuration.

Audience

This document is intended for IT architects, engineers, and administrators who are responsible for configuring and

deploying the SAP HANA platform in a VMware virtualization environment. It assumes that the reader has a basic

knowledge of FlashStack, Cisco UCS, VMware vSphere, Linux, and SAP HANA TDI scenarios.

Document reference

Please refer the Cisco® Validated Design document FlashStack for SAP HANA TDI published in February 2017 for

deeper understanding of FlashStack converged infrastructure design principles and configuration. This document

builds on this design and addresses the virtualization scenario, highlighting general best practices.

The best practices documented here are in line with VMware’s Best Practices Guide for SAP HANA and serves as

a reference for implementation on FlashStack converged infrastructure.

Test environment

Table 1 lists the hardware and software versions used in the test environment described in this document.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 17

Table 1. Test environment details

Layer Device Image

Compute Cisco UCS 6332-16UP Fabric Interconnect pair Release 3.1(2b)

Cisco UCS C460 M4 Server (3 nodes) Release 3.1(2b)

Cisco UCS Virtual Interface Card (VIC) 1385 (1per C460 M4 node) Release 3.1(2b)

Network Cisco Nexus® 9372PX-E Switch pair Release 7.0(3)I2(4)

Storage Cisco MDS 9148S 16G Multilayer Fabric Switch pair Release 7.3(0)DY(1)

Pure Storage FlashArray //m50 R2 Release 4.8.3

Software Cisco UCS Manager Release 3.1(2b)

VMware vSphere ESXi Release 6.5.0

VMware vCenter Release 6.5.0

Reference architecture

Figure 1 shows the various layers of architecture, from the storage layer to the application layer. It shows the

possible constituents of each layer for FlashStack converged infrastructure.

FlashArray //m series arrays make up the storage layer. Cisco MDS 9100 Series Multilayer Fabric Switches and

Cisco Nexus 9000 Series Switches at the network layer cater to Fibre Channel and Ethernet traffic respectively.

Cisco UCS C-Series Rack Servers or B-Series Blade Servers provide the compute. SAP HANA is installed in

virtual machines running either the SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL) OS

hosted on the VMware ESXi virtualization platform.

Figure 1. Architecture overview: Layered representation

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 17

Virtualized SAP HANA can be deployed in both scale-up and scale-out system configurations.

Storage layer

The Pure Storage Purity Operating Environment automatically handles the tuning, encryption, solid-state disk

(SSD) wear leveling, and data resiliency (among other operations) at the array level. You only need to name the

volume (the target for the datastore), enter the size, and attach the volume to the appropriate ESXi host groups.

An important benefit of Pure Storage FlashArray//m is that it enables customers to design their SAP HANA

environment datastores with a logical approach rather than constraining them to the limitations of virtual machines

per datastore. However, it is a good practice to segregate the instance’s OS boot disks and SAP HANA disks

(data, log, and hanashared) in separate datastores, at a minimum. For example, Figure 2 shows a virtualized SAP

HANA scale-up system cisvhana1 with 512 GB of RAM configured with a 100-GB OS boot disk (including a

/usr/sap partition) along with the configuration file in one datastore (VMboot-Datastore). It also includes disks for

/hana/data (768 GB), /hana/log (256 GB), and /hana/shared (512 GB) in another datastore (vHANA-Datastore).

Figure 2. Datastore selection and disk formatting

A single datastore hosting SAP HANA disks (data, log, and shared) would also suffice to support multiple

virtualized SAP HANA instances for most use cases. Configuring separate datastores for data (requiring higher

read performance) and logs (requiring higher write performance), though desirable, may not be necessary because

the back-end all-flash array provides a uniform performance-class I/O subsystem, managed by the Purity

Operating Environment.

Server and network layers

This section describes best practices for the server and network layers.

Multipath configuration

VMware vSphere ESXi provides the platform for the virtualized SAP HANA system virtual machines. ESXi uses its

Native Multipathing Plugin (NMP) architecture to manage I/O multipathing to underlying SAN storage volumes.

FlashArray has an active-active front-end controller architecture. FlashArray volumes are preferably claimed, by

default, by the Storage Array Type Plugin (SATP) for Asymmetric Logical Unit Access (ALUA) devices and use

round-robin (RR) path selection policy (PSP). That is, they use VMW_PSP_RR with an I/O operations limit of 1 in

order to alternate logical paths after every I/O operation. Refer to VMware Docs for more information.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 17

If this setup is not available in a particular ESXi release, the same result can be achieved by running the following

command on each ESXi host prior to the presentation of FlashArray devices:

#esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "PURE" -M "FlashArray" – P

"VMW_PSP_RR" -O "iops=1"

Power management

A Cisco UCS service profile defines a single server and its storage and networking characteristics. It is a logical

representation of a server that encapsulates identity, configuration, and connectivity information. Service profiles

also include operational policy information such as BIOS parameters, boot paths, and boot order.

SAP HANA workloads are sensitive to latency. Because achieving the lowest possible latency and saving power on

the hosts are mutually exclusive, the recommended approach is to disable any form of power management to aid

applications when performance needs are more important. In this case, you should disable the C-state and

enhanced C-state (C1E) power management options.

Additionally, set the ESXi host’s hardware power management to the high-performance mode (Figure 3).

Figure 3. VMware ESXi host Power Management settings

Figure 4 shows optimum BIOS parameter settings for best performance in a virtualization scenario.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 17

Figure 4. Cisco UCS service profile BIOS policy Processor settings

Hyperthreading and non-uniform memory access management

Hyperthreading and non-uniform memory access (NUMA) management are especially important for configurations

with multiple virtual machines. SAP HANA workloads are sensitive to memory latency with relatively lower

processor utilization. For such workloads, the recommended approach is to use hyperthreading with fewer NUMA

nodes. You also should size the virtual machines to stay within a NUMA node or within the least number of NUMA

nodes, as best as possible, to optimize performance.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 17

To achieve this setup, implement the configuration parameter numa.PreferHT=1 on a per-host basis (Figure 5).

Alternatively, you can implement the parameter numa.vCPU.preferHT=TRUE on each virtual machine. Refer to

VMware KB 2003582 for more details.

Figure 5. VMware ESXi host: Advanced system settings for numa.PreferHT

Fibre Channel connectivity

The FlashArray//m series are block-only arrays. The ESXi hosts should be configured to boot from SAN. A

redundant pair of virtual host bus adapters (vHBAs)—one each for SAN fabric A and SAN fabric B defined in the

service profile—serve as the initiators (Figure 6). A 10-GB volume from the array can be presented to the host and

used as its ESXi boot disk (Figure 7).

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 17

Figure 6. Cisco UCS service profile template: Storage > vHBA settings

Figure 7. Pure Storage GUI: Properties for ESXi host boot from SAN disk

Network considerations

SAP categorizes the HANA system networks into three logical communication zones (Figure 8):

● Client zone: External communication networks (for example, for application server and client network

connectivity)

● Internal zone: Internal communication networks (for example, for internode communication in a scale-out

system and replication traffic in a distributed configuration)

● Storage zone: All storage-related networks (for example, for the Network File System (NFS)–based

/hana/shared network and backup network and for NFS-based data and log file system access when NFS is

used)

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 17

Figure 8. SAP HANA TDI network requirements (source: SAP SE)

Table 2 summarizes the various use cases that these networks serve and their relevance to a virtualized SAP

HANA solution on FlashStack.

Table 2. SAP HANA TDI and FlashStack network requirements mappings

Name Use case FlashStack virtualized SAP HANA solution relevance

Minimum bandwidth requirements

Client zone

Application server network Communication between SAP application server and database

Required 1 or 10 Gigabit Ethernet

Client network Communication between user or client application and database

Required 1 or 10 Gigabit Ethernet

Internal zone

Internode network Node-to-node communication within a scale-out configuration

Required for scale-out configurations only

10 Gigabit Ethernet

System replication network SAP HANA system replication Optional To be determined with customer

Storage zone

Backup network Data backup Optional 10 Gigabit Ethernet or 8-Gbps Fibre Channel

Storage network Communication between nodes and storage

● Optional for scale-up systems

● Required for NFS-based /hana/shared file system access in scale-out scenario

10 Gigabit Ethernet

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 17

Infrastructure-related networks

Administration and VMware High Availability (HA) network

SAP HANA system administration and VMware HA management

Required 1 Gigabit Ethernet

VMware vMotion network Resource utilization balancing for a vSphere cluster

Required when using multiple ESXi hosts in a VMware Distributed Resource Scheduler (DRS)–enabled cluster

10 Gigabit Ethernet

To determine the number of virtual network information cards (vNICs) to define when creating the service profiles

to support both bandwidth and failover requirements, map the use-case information using the matrix in Table 2.

Note the following:

● Traffic management for the various networks can be performed at the virtual machine level using VLANs

while ensuring consistency with the definitions on the data center uplink switch.

● For network bandwidth management when using a single virtual switch (vSwitch) to consolidate all network

traffic or when using a separate vSwitch for each zone to segregate traffic based on the required bandwidth,

be sure that the optimum number of load-balanced uplinks are assigned.

● For the internal zone, storage zone, and vMotion network, the recommended approach is to configure

jumbo frames end-to-end.

Virtual machine, OS and application layers

This section presents best practices for the virtual machine, OS, and application layers.

Virtual machine configuration

The recommended approach is to configure the cores per socket to match the actual hardware configuration. For

example, configure 22 cores per socket for Intel® Xeon

® processor E7-8880 v4, 24 cores per socket for Intel Xeon

processor E7-8890 v4, and 28 cores per socket for Intel Xeon Platinum 8180 and 8176 processors (Figure 9).

Figure 9. Setting cores per socket

Be sure that the vCPU hot plug is not selected because it disables the virtual NUMA (vNUMA) optimization feature

(Figure 10).

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 17

Figure 10. CPU Hot Plug setting

The recommended approach is to reserve CPU and especially the memory resources for production virtual

machines. Select the “Reserve all guest memory (All locked)” option (Figure 11).

Figure 11. Virtual machine memory reservation setting

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 17

Use paravirtual adapter VMXNET 3 type vNICs to configure the network interfaces for all the networks defined

(Figure 12).

Figure 12. vNICs adapter type selection

Use a dedicated Small Computer System Interface (SCSI) controller of type VMware Paravirtual for each data, log,

and hanashared disk to implement parallel I/O processing by separating the disk I/O streams. Also use eager-

zeroed thick provisioning for SAP HANA data and log disks (Figure 13).

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 17

Figure 13. SCSI controller type and disk provisioning type selection

Operating system

To configure optimal settings for running SAP HANA on SLES or RHEL, follow the respective SAP notes that

provide the SAP HANA database recommended OS settings guidelines as you would do for bare-metal

implementations. See the “For more information” section of this document for all relevant SAP Notes.

ESXi has its own I/O scheduling mechanisms that not only makes guest OS IO scheduling redundant but also adds

to unnecessary overhead. The recommended approach is to disable I/O scheduling at the virtual machine level.

Append elevator=noop to the kernel boot arguments.

Application layer

Both scale-up and scale-out configurations are possible in the virtualized environment, as they are in bare-metal

scenarios. Note the following points when implementing these configurations on FlashStack:

● In a scale-up system configuration, /hana/shared can also be a virtual machine disk (VMDK) created from

the same datastore as the one that hosts the data and log disks of the SAP HANA instance because it is

local to that instance. It does not necessarily be a NFS mount.

● In a scale-out configuration, /hana/shared must be accessible by all nodes of the scale-out cluster

concurrently, and at all times needing it to be a NFS mount. FlashArray //m series arrays are block-only

arrays. In a bare-metal scenario, this setup is achieved by configuring the scale-out nodes’ OS in a high-

availability cluster with /hana/shared configured as an Oracle Cluster File System (OCFS2) cluster file

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 17

system resource. (Refer to the Cisco Validated Design for FlashStack for SAP HANA TDI for more

information.) However, a VMware-supported scale-out solution requires the SAP HANA shared file system

to be NFS only. Other solutions—such as OCFS and IBM General Parallel File System (GPFS)—are not

supported by VMware. To address this requirement, you can configure a dedicated virtual machine as an

NFS server and export a locally mounted disk as /hana/shared to the cluster nodes. The high availability of

the NFS server virtual machine can be ensured by activating the vSphere HA and DRS features and, in

extreme cases, enabling VMware Fault Tolerance (FT), taking into account proper sizing considerations.

Sizing

This section provides sizing inputs in terms of the virtual machine’s memory to virtual CPU (vCPU) ratio and the

number of virtualized SAP HANA instances and nodes supported by FlashArray //m50 R2.

Compute

The CPU generation platform and SAP’s guidelines regarding the memory range supported for a particular socket

count form the basis for sizing virtualized SAP HANA instances. The vCPU and virtual RAM (vRAM) maximums for

a specific vSphere ESXi version also influence the sizing. Additionally, some memory overhead should be

considered because ESXi needs some for its own kernel and for hosting virtual machine memory pages. Refer to

VMware Docs for more information.

With Intel Xeon Platinum platform, SAP allows memory configurations of 128 GB to 3 TB on four-way server

systems for SAP Business Warehouse on HANA (BWoH). It allows up to 6 TB for SAP Suite on HANA (SoH)

workloads. The matrix in Table 3 provides a reference for technically feasible virtual machine configurations with

the current Intel Xeon Platinum platform (supported only on Cisco UCS M5 servers) and previous Intel Xeon

processor generations.

Table 3. Virtual machine sizing: vCPU and vRAM with processor platform matrix

vCPU (with hyperthreading on) vRAM

Intel Xeon E7 v4 8880 Intel Xeon E7 v4 8890 Intel Xeon Platinum 8180 and 8176

BWoH SoH BWoH SoH BWoH SoH

≤SPS10 ≥SPS11 ≤SPS10 ≥SPS11 ≤SPS11 ≥SPS12

22 22 22 24 24 24 24 28 28 32

22 22 22 24 24 24 24 28 28 64

22 22 22 24 24 24 24 28 28 128

22 22 22 24 24 24 24 28 28 256

44 44 22 48 48 24 24 28 28 384

66 44 44 72 48 48 24 28 28 512

88 66 44 96 72 48 48 56 28 768

110 88 66 120 96 72 48 84 28 1024

110 88 120 96 72 112 56 1536

110 120 96 84 2048

120 112 3072

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 17

Storage

In the test environment with FlashArray //m50 R2, tests were run with a single datastore for SAP HANA data and

log files to ascertain the number of virtualized SAP HANA instances that the array could support, satisfying the

SAP HWCCT tool TDI key performance indicators (KPIs).

Table 4 summarizes the results when tests were run in the lab environment with one datastore for both HANA

persistence partition constituents. The recommended approach is to perform case-specific HWCCT-based tests to

verify that the system meets the TDI KPIs.

Table 4. Array sizing test results with fsperf parameters used

FlashArray //m series array Number of datastores for SAP HANA data and log disks

Number of virtualized SAP HANA instances supported

//m50 R2 01: One datastore hosting both data and log disks of all the virtualized SAP HANA instances.

Up to 12

For more information about the HWCCT based fsperf parameters, refer to SAP Note 1943937.

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 17

For more information

Certified SAP HANA Hardware Directory

● Certified SAP HANA Hardware Directory: Enterprise Storage

SAP HANA TDI documentation

● SAP HANA TDI: Overview

● SAP HANA TDI: FAQ

● SAP HANA TDI: Storage Requirements

● SAP HANA TDI: Network Requirements

SAP Notes

● SAP Note 2161991: VMware vSphere configuration guidelines

● SAP Note 1788665: SAP HANA Support for virtualized / partitioned (multi-tenant) environments

● SAP Note 2393917: SAP HANA on VMware vSphere 6.5 in production

● SAP Note 1943937: Hardware Configuration Check Tool - Central Note

● SAP Note 2235581: SAP HANA: Supported Operating Systems

Other links

● SAP Community Network: SAP HANA TDI on Cisco UCS and VMware vSphere

● SAP Document Archive link: https://archive.sap.com/documents/docs/DOC-60312

● Cisco UCS: Design Zone for SAP Applications (technical documentation)

● Cisco UCS: Data Center Solutions for SAP (customer references)

● FlashStack converged infrastructure: FlashStack Virtual Server Infrastructure Design Guide for VMware

vSphere 6.0 U2

● FlashStack converged infrastructure: Cisco Validated Design for FlashStack VDI use case

● VMware blog: SAP on VMware updates

Printed in USA C11-739549-00 08/17