SAP HANA TDI Configuration Guidelines for Lenovo System x ...

EMC WHITE PAPER

SAP HANA TDI Configuration Guidelines for Lenovo System x Servers and EMC VMAX Storage Using IBM GPFS and XFS File System

ABSTRACT

This white paper describes the advantages of using EMC® VMAX® enterprise storage and Lenovo System x servers in an SAP HANA TDI environment. Included are best practices and configuration guidelines for EMC VMAX storage and Lenovo System x Server with IBM GPFS and XFS file systems in a SAP HANA TDI implementation that meets or exceeds all SAP performance requirements and ensures the highest availability for database persistence on disk.

April 2015

2 SAP HANA TDI Configuration Guidelines for Lenovo System x Servers and EMC VMAX Storage Using IBM GPFS and XFS File System

Disclaimer

This document outlines a general product direction and should not be relied on in making a purchase decision. This document is not subject to your license agreement or any other agreement with EMC or SAP. EMC or SAP has no obligation to pursue any course of business outlined in this document or to develop or release any functionality mentioned in this document. This document and the strategy and possible future developments of EMC or SAP are subject to change and may be changed by EMC or SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. EMC and SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by EMC or SAP intentionally or grossly negligent.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of EMC and SAP AG.

Copyright © 2015 SAP. All Rights Reserved.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in other countries.

See http://www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

Lenovo, the Lenovo logo, and For Those Who Do are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. These and other Lenovo trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by Lenovo at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of Lenovo trademarks is available on the Web at HUhttp://www.Lenovo.com/legal/copytrade.html The following terms are trademarks of Lenovo in the United States, other countries, or both: BladeCenter® eX4™ eX5™ eXFlash™ FlashCache™ Flex System™ Lenovo® MAX5™ RackSwitch™ Lenovo(logo)® ServeRAID™ System x® X5™ X6™ The following terms are trademarks of other companies: Intel, Intel Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. IBM, IBM GPFS are trademarks or registered trademarks of IBM Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries or both.

Copyright © 2015 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

Part Number H14167

http://www.sap.com/corporate-en/legal/copyright/index.epx


Table of Contents

Executive summary.................................................................................................. 5 Purpose .............................................................................................................................. 6 Scope ................................................................................................................................. 6 Audience ............................................................................................................................ 7

EMC VMAX and Lenovo ............................................................................................ 7 Overview ............................................................................................................................ 7

Landscape optimization and data center readiness ........................................................ 7 EMC VMAX enterprise storage ............................................................................................. 7

EMC Enterprise Storage offerings in SAP HANA TDI scenarios ......................................... 7 Disaster tolerance using EMC VMAX SRDF .................................................................... 10

Lenovo ............................................................................................................................. 10 Lenovo System x offerings in SAP HANA TDI compute area ........................................... 10 Lenovo System x enterprise solution services for SAP HANA ......................................... 11

System x UEFI settings for maximum performance in an SAP HANA environment ..... 12 Lenovo System x UEFI settings .......................................................................................... 12

Intel Xeon E5 and E7 processor based systems ............................................................ 13 Intel Xeon E5 V2 processor based systems ................................................................... 13 Intel Xeon E7 V2 processor based systems ................................................................... 14 Intel Xeon E5 V3 processor based systems ................................................................... 14

Setting C-States ................................................................................................................ 14 Using the Advanced Settings utility .................................................................................. 16 Using the System x UEFI boot screen ................................................................................ 16 SAP HANA System Check .................................................................................................. 16

IBM GPFS installation and configuration ................................................................. 17 Network configuration ...................................................................................................... 17 Customize /etc/hosts file ................................................................................................. 18 SSH configuration ............................................................................................................ 18 Install GPFS packages ...................................................................................................... 19 Update to the latest GPFS maintenance release ................................................................ 19 Build the GPFS portability layer ........................................................................................ 20 Prepare GPFS configuration .............................................................................................. 20 GPFS cluster configuration ................................................................................................ 21

XFS configuration for SAP HANA scale-out using storage connector API ................... 26 Network configuration ...................................................................................................... 26 XFS file system configuration for 2+1 HANA scale out ....................................................... 26 SAP HANA global.ini file ................................................................................................... 29 HANA DB installation ........................................................................................................ 29


HA tests (Scale-out node failover test).................................................................... 36 Worker node failover ........................................................................................................ 36 Master/Worker node failover – Hardware failure simulation ............................................. 37

Appendix A ........................................................................................................... 37 HANA compute node details ............................................................................................. 37 SAP HANA database version used for the tests ................................................................. 38 SAN details ...................................................................................................................... 38

Appendix-B ........................................................................................................... 38 Lenovo compute building blocks for SAP HANA TDI .......................................................... 38 General capacity considerations for VMAX storage in TDI .................................................. 39 QLogic Fibre Channel Adapters for Lenovo System-X Servers ............................................ 40

Management software .................................................................................................. 40 HWCCT ............................................................................................................................. 41

Resources ............................................................................................................. 41


Executive summary In today's fast-paced world, you need to know what is going on with your business, your marketplace, and your customers so you can respond quickly. Many enterprises are looking to obtain real business insights made possible with SAP HANA. However, the move to SAP HANA can be complex, so choosing your business and technology partners wisely is an important first step of this journey.

Lenovo and EMC® are working closely with SAP to provide you with seamless, secure, context-aware experiences for your SAP environment. For SAP HANA, Lenovo and EMC offer customers a complete set of fully-integrated solutions and services with Lenovo System x servers and EMC Enterprise Storage for customers wanting to deploy their own SAP HANA environments.

By default, a SAP HANA appliance includes integrated storage, compute, and network components. SAP has defined key performance indicators for storage subsystems, such as I/O throughput and latency, ensuring that the overall infrastructure forms a HANA-ready environment.

You can also deploy SAP HANA appliances using a Tailored Data Center Integration (TDI) approach. With this approach, you can connect SAP certified servers to existing enterprise storage systems. This reduces time-to-value, risk, and costs for an overall HANA adoption. Using this approach, you can leverage existing data center processes, as well as storage infrastructure.

The SAP HANA configuration outlined in this white paper is based on Lenovo System x servers and EMC VMAX® enterprise storage system. The document covers SAP HANA scale out architecture by using IBM General Parallel File System (GPFS) and XFS in SAP HANA scale out implementation. The solution described in this white paper combines the technologies of EMC and Lenovo to have a best in-class SAP HANA TDI implementation.

Lenovo and EMC engineering performed many performance tests using the SAP HANA hardware configuration and check tool (HWCCT) on an EMC VMAX enterprise storage system, Lenovo System x severs using IBM GPFS and XFS. Based on these tests, this white paper describes scale-out SAP HANA configuration that meets or exceeds all SAP performance requirements and ensures the highest availability for database persistence on disk. This is one option that many that customers can define to build a HANA database. Refer to Appendix A for more information on EMC VMAX storage and Lenovo System x servers used in these tests.

SAP recommends that TDI customers run the HWCCT tool in their environment to ensure that the customer specific HANA TDI implementation meets the SAP performance criteria.

This technical white paper gives you the details of how you can integrate EMC VMAX Storage and Lenovo System x servers with SAP HANA TDI using SAP specifications for enterprise storage. This paper describes key solution components, configuration, and


implementation details plus provides information on the unique architectural advantages of EMC VMAX enterprise storage and Lenovo System x servers for TDI. This paper includes step-by-step instructions to configure IBM GPFS files system and XFS file system on a VMAX array for SAP HANA scale-out configuration in TDI deployments on Lenovo System x servers.

Purpose

To meet customer’s needs for SAP HANA, Lenovo offers them two distinct delivery alternatives: complete appliance delivery or TDI. The Lenovo System Solution for SAP HANA is a fully integrated, optimized appliance solution that offers the performance and robustness needed to handle the challenges of a combined transaction and analytics platform. The Lenovo System Solution for SAP HANA is an easy-to-implement, standardized, highly optimized solution with convenient preconfigured hardware and preinstalled software. The solution is fully validated by Lenovo and SAP.

However, when a more customized data center solution is desired as an alternative to appliance-based solutions, Lenovo and EMC offers solutions for SAP HANA TDI. The EMC VMAX Storage integration is fully certified and authorized by SAP to give you the flexibility to specify key components within your SAP HANA infrastructure. Although customers take responsibility for the validation of their SAP HANA environment with TDI, this approach offers a few key benefits:

• Reduced costs through usability of existing hardware components

• Mitigated risk and improved time to value by enabling existing operation and management processes

Scope

This white paper describes the following:

• IBM GPFS installation and configuration for SAP HANA scale out in TDI.

• XFS file system configuration for SAP HANA scale out.

• SAP HANA High Availability using SAP HANA storage connector API and EMC VMAX shared storage and Lenovo System x servers.

• Key benefits of using EMC VMAX enterprise storage and Lenovo System x servers with SAP HANA.

Note: This paper primarily discusses SAP HANA scale out architecture and high availability feature tests on IBM GPFS and XFS, but a SAP HANA scale up (single node) for Business Warehouse (BW) on HANA and Suite on HANA (SoH) implementation is a tested and validated configuration by Lenovo and EMC.


Audience

This whitepaper is intended for Lenovo and EMC system integrators, systems and storage administrators, partners, customers, members of Lenovo lab services and EMC professional services involved in a SAP HANA TDI implementation.

EMC VMAX and Lenovo This section provides information on EMC VMAX enterprise storage and Lenovo System x servers.

Overview

The Lenovo System x server portfolio for SAP HANA offers unprecedented scalability. Large memory configurations are available for SAP Business Suite, powered by SAP HANA. And for SAP Business Warehouse and data mart implementations, you can easily create multi-node scale-out configurations by networking multiple nodes together enabling support for larger SAP HANA memory sizes. This modular approach enables you to invest in an optimized solution for SAP HANA and grow your infrastructure as your SAP HANA environment grows. In addition, solution from Lenovo System x and EMC VMAX integrates high availability and disaster recovery with an automatic failover feature by using EMC SRDF® (Symmetrix Remote Data Facility), IBM GPFS file system, or XFS file system.

Landscape optimization and data center readiness

Choose from a pre-configured solution or a custom Tailored Datacenter Integration (TDI) solution. Effectively protect your data with integrated high availability and synchronous disaster recovery capabilities using IBM General Parallel File System (GPFS) software and EMC SRDF. Additionally, VMware virtualization technology can be used for optimizing deployment and management of your SAP HANA application environment using virtual machines via a single, SAP HANA certified system.

EMC VMAX enterprise storage

EMC Enterprise Storage offerings in SAP HANA TDI scenarios

The specific SAP HANA I/O workload requires special considerations for the configuration of persistence for data and log volumes on a shared EMC VMAX storage array.

The VMAX system delivers the high performance requirements for the persistent storage of a SAP HANA database, without impacting other applications concurrently using the same array.

The I/O workload on SAP HANA has two major components:

• Random I/Os on the data volume

• Sequential I/Os on the log volume


SAP HANA uses different block sizes for log volumes and data volumes.

• I/O for data volumes uses 4 K, 16 K, 64 K, 1 M, 16 M, and 64 M block sizes.

• I/O for log volumes uses 4 K, 16 K, and 1 M block sizes.

EMC VMAX scalability

In an EMC VMAX 10K, VMAX 20K, or VMAX 40K VMAX array, the scalability of SAP HANA primarily depends on the number of available engines in the array. Table 1 shows the VMAX models and the estimated maximum number of HANA worker nodes that can be connected according to the number of available engines.

Table 1. EMC VMAX 10K, VMAX 20K and VMAX 40K scalability


When EMC Symmetrix Remote Data Facility (SRDF) is used for SAP HANA storage replication, a reduced number of front-end FA-ports will be available and the number of HANA worker nodes which can be connected to the array must be adjusted accordingly.

EMC VMAX3 scalability

In an EMC VMAX3 100K, 200K, 400K array, the scalability of SAP HANA primarily depends on the number of available engines in the array. Table 2 shows the VMAX3 models and the estimated maximum number of HANA nodes that can be connected according to the number of available number of engines.

Table 2. EMC VMAX3 100K, VMAX 200K and VMAX 400K scalability

If you use SRDF for SAP HANA storage replication, a reduced number of front-end FA-ports are available, and the maximum number of HANA worker nodes that can be connected to the array must be adjusted accordingly.


Disaster tolerance using EMC VMAX SRDF

Leveraging EMC VMAX SRDF and EMC TimeFinder® to protect the entire SAP landscape

Storage replication offers advantages over application-based disaster recovery solutions where only the database content is replicated to the secondary site. Leveraging SRDF provides a disaster tolerance solution for the entire SAP landscape. EMC VMAX SRDF consistency technology can be leveraged to include all relevant HANA and non-HANA volumes, even across multiple arrays at each site, ensuring fast recovery of the entire SAP landscape, not only the database.

Figure 1. EMC VMAX SRDF for disaster recovery

EMC TimeFinder can be used for a rapid recovery of the database and to augment remote replication for specific use cases.

Lenovo

Lenovo System x offerings in SAP HANA TDI compute area

• Choice of base memory size: 128 GB, 256 GB, 512 GB, 1 TB, or 2 TB.

• Scalable memory support up to 6 TB production and 12 TB non-production for SAP Business Suite powered by SAP HANA.

• Flex Node support enables you to maximize your IT resources and provide flexible capacity planning for non-production environments allowing you to partition your 8-socket x3950 X6 (Lenovo MTM: 6241) SAP HANA System x as one 8-socket system or two x3850 X6 4-socket systems.


• Software: IBM General Parallel File System (GPFS) on x86 Single Server license. To use these server models in a scale-out cluster, additional GPFS Server and GPFS FPO licenses will need to be purchased.

The new System x X6 enterprise servers with Enterprise X Architecture are designed to provide faster performance and support for more memory. Using self-contained compute books and Intel Xeon E7 product family processors, the System x Solution for SAP HANA on X6 is purpose built to support large-scale SAP HANA deployments by incorporating, Up to 120 cores per server for maximum CPU performance and proven expansion capacity to scale up to a 100 TB SAP HANA system.

Flex System x880 is the world’s first 8-socket Intel Xeon processor-based blade server. It provides up to 2x the capacity of competitive blade servers and offers scalability from 2- to 4- or 8-sockets enabling you to scale up processors and memory for large SAP Business Suite on HANA databases add IT resource to your environment as your demands increase. It offers high-end server resiliency and availability to help maximize application uptime for critical SAP database and business applications. I addition, it provides the ability to integrate and optimize compute, storage and networking resources with simplified management using the no-comprise building blocks of the Flex System infrastructure.

Figure 2. Flex System x880 X6 X3950 X6 8 Socket HANA compute Node: (MTM: 6241)

Delivered on Flex System x880 X6 compute nodes that provide leading performance and scalability, SAP HANA efficiently processes and analyzes massive amounts of data by packaging SAP’s use of in-memory technology, columnar database design, data compression and massive parallel processing together with essential tools and functionality such as data replication and analytic modeling. Using SAP HANA as transactional single-node RDBMS for e.g., SAP Business Suite, customers can benefit from the in box memory scalability of the X6-technology.

Lenovo System x enterprise solution services for SAP HANA

The System x Lab Services team helps clients worldwide with deployment of System x solutions and technologies. Locally or globally, Lenovo offers installation, configuration, integration, migration and other professional services in support of


System x servers and solutions. Whether training staff at customer site to get up to speed, performing health check services for existing environment, or architecting the next design, Lenovo has offerings and expertise to meet customer’s needs.

Lab services

Lab Services will help you configure, validate hardware and software stack (OS, General Parallel File System (GPFS), and SAP), configure networking, provide basic skills transfer and post-install documentation for SAP HANA on Lenovo System x hardware. Learn more

SAP HANA managed services

This service offers constant remote monitoring of HANA hardware and software. It also includes monitoring of HANA software stack (OS, GPFS, and SAP), ongoing health checks, scheduled upgrades, ownership and resolution of related problems, and maintenance of records. Learn more

SAP HANA health check services

This offering provides health check services for SAP HANA. It includes on-site assessment of HANA software stack (OS, GPFS, SAP), networking, solution configuration, and post-install documentation on IBM System x hardware. This service includes maintaining and updating firmware and software for the SAP HANA solution on IBM System x hardware to SAP and IBM best practices levels. Learn more

System x UEFI settings for maximum performance in an SAP HANA environment

Lenovo System x UEFI settings

In order to obtain the most optimal performance for the SAP HANA configuration, it is necessary for some of the System x UEFI settings to be modified from their original defaults. It is recommended that you change the following parameters to the described settings based upon the System x type you are installing while leaving all others at their default values.

Please note that if you update any of the firmware of the System x machine, it may occur that the UEFI settings will be reset. Therefore, it is recommended that you recheck the UEFI settings after each firmware update and change the values as necessary to the documented values.

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=SP&infotype=PM&appname=STGE_QL_QL_USEN&htmlfid=QLS12367USEN&attachment=QLS12367USEN.PDF




Intel Xeon E5 and E7 processor based systems

Table 3. Required UEFI settings for System eX5 servers

Parameter Value ASU Tool Setting x3690 x3850 Hyper Threading Enable UEFI.HyperThreading

Operating Mode Performance Mode UEFI.OperatingMode

QPI Link Speed Max Performance UEFI.QPISpeed

Memory Patrol Scrub Enable UEFI.PatrolScrub

Intel Xeon E5 V2 processor based systems

Table 4. Required UEFI settings for System M4 servers based in Intel Xeon V2

Parameter Value ASU Tool Setting x3500 x3550 x3650 X*40 HS23 Choose Operating Mode

Custom Mode

OperatingModes.ChooseOperatingMode

Active Energy Manager

Capping Disable

Power.ActiveEnergyManager

Platform Controlled Type

Max Performance

Power.PlatformControlledType

Workload Configuration

I/O sensitive Power.WorkloadConfiguration

Memory Power Management

Automatic Memory.MemoryPowerManagement

C1 Enhanced Mode

Disable Processors.C1EnhancedMode

Direct Cache Access (DCA)

Enable Processors.DirectCacheAccessDCA



Table 5. Required UEFI settings for System x6 servers based in Intel Xeon V2

Parameter Value ASU Tool Setting x3850 x3950 x280 x480 x880 Choose Operating Mode

Custom Mode OperatingModes.ChooseOperatingMode


Capping Disable



Max Performance






C1 Enhanced Mode

Disable Processors.C1EnhancedMode

Enable SMK Disable Processors.EnableSMK




Table 6. Required UEFI settings for System M5 servers based in Intel Xeon V3

Parameter Value ASU Tool Setting x3500 x3550 x3650 x*40 nxM5 Choose Operating Mode

Custom Mode OperatingModes.ChooseOperatingMode


Capping Disable



Max Performance






C1 Enhanced Mode Disable Processors.C1EnhancedMode



Setting C-States

As recommended in the SAP Notes 1824819 – SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP2 and 1954788 – SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP3 and additionally described in the IBM RETAIN Tip H20700034 – Linux Ignores C-State Settings in Unified Extensible Firmware Interface (UEFI), the control (’C’) states of the


Intel processor should to be turned off for the most reliable performance of SAP HANA.

By default C-States are enabled in the UEFI due to the fact that we set the processor to Custom Mode. With C-States being turned on you might see performance degradations with SAP HANA. We recommend turning off the processor C-States using the Linux kernel boot parameter:

processor.max_cstate=0

The Linux kernel used by SAP HANA includes a built-in driver (’intel_idle’) which will ignore any C-State limits imposed by Basic Input/Output System (BIOS)/Unified Extensible Firmware Interface (UEFI) when it is active. This driver may cause issues by enabling C-States even though they are disabled in the BIOS or UEFI. This can cause minor latency as the CPUs transition out of a C-State and into a running state. This is not the preferred state for the SAP HANA configuration and must be changed.

To prevent the ’intel_idle’ driver from ignoring BIOS or UEFI settings for C-States, add the following Start parameter to the kernel’s boot loader configuration file:

intel_idle.max_cstate=0

Append both parameters to the end of the kernel command line of your boot loader (/boot/grub/menu.lst) and reboot the server. You can use the SuSE yast tool and configure the same as well.

Figure 3. Setting C-States example


Using the Advanced Settings utility

Lenovo has an Advanced Settings Utility tool, ASU, for Lenovo System x Servers. This tool can be installed on top of a running operating system in order to view and change settings in the System x UEFI firmware. This tool can be downloaded at https://www-947.IBM.com/support/entry/portal/docdisplay?lndocid=LNVO-ASU

It is recommended that you use this tool to properly set the UEFI settings.

1. Install the ASU tool.

To install from rpm file:

a) Copy the rpm to a directory on the hard drive.

b) Run one of the following commands as the root user:

To install if the package is not already installed:

# rpm -i <ASU distribution>.rpm

To install or update the package, removing other versions:

# rpm -U <ASU distribution>.rpm

2. Display current values of the running system.

# /opt/IBM/toolscenter/asu/asu64 show

3. Save changed settings to file.

# /opt/IBM/toolscenter/asu/asu64 show --kcs > /tmp/myasu.txt

4. Modify values with a text editor.

5. Store to UEFI

# /opt/IBM/toolscenter/asu/asu64 set < /tmp/myasu.txt

Using the System x UEFI boot screen

If you wish to change the settings before you install the operating system, or need to modify the settings after an update of the System x firmware, then you may also enter the System x UEFI boot screen to manually change the settings at boot time. In order to enter the boot screen, you must press the <F1> function key while booting the server. When the setup screen is displayed, you may enter each of the above settings from their respective groupings.

SAP HANA System Check

An optional, but very important tool is the SAP HANA System Check tool, which ensures that SAP HANA has been installed on valid and certified hardware. This tool does not check whether the system can perform adequately; it only ensures that all the hardware used is acceptable to the standards set out by SAP SE for its SAP HANA software.

https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=LNVO-ASU

https://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=LNVO-ASU


To run this tool, you will need to enter the directory DATA_UNITS/HDB_SERVER/server and execute the following command:

HANAtdi:{HANA dump}/DATA_UNITS/HDB_SERVER_LINUX_X86_64/server# python HANAHwCheck.py

Using this tool can give you an initial overview if those settings, as checked by SAP, are correct and usable for SAP HANA. In case this tool shows errors, or if you have questions regarding its output, open an SAP Customer Message on SAP Service Marketplace to obtain help from SAP.

IBM GPFS installation and configuration

Network configuration

Before you start GPFS cluster configuration, you need to gather the networking details listed in the following table.

Table 7. GPFS network configuration requirements in a SAP HANA scale out implementation

Network Description

IP Addresses Host Name Net Mask Gateway

SAP HANA Internodes Communication

192.168.20.X

HANAnode0X 255.255.255.0 None

GPFS Internodes Communication

192.168.10.X GPFSnode0X 255.255.255.0 None

IMM Customer Defined

Customer Defined

Customer Defined Customer Defined

Customer Network Customer Defined

Customer Defined

Customer Defined Customer Defined

X = Node number


SAP HANA and GPFS inter-node communication private IPs to be configured on the 10G NIC ports.

Example: In a 3-node (2*worker + 1*standby) cluster setup, you will have the following network configurations:

Node 1 192.168.20.1 (HANAnode01), 192.168.10.1 (GPFSnode01)



Note: Mandatory to use 10G NIC to meet the SAP defined KPIs in TDI network.

Customize /etc/hosts file

Add GPFS host names and IP addresses of all cluster nodes to the /etc/hosts file.

192.168.10.101 GPFSnode01 GPFSnode01

192.168.10.102 GPFSnode02 GPFSnode02 192.168.10.103 GPFSnode03 GPFSnode03

: : :

192.168.10.116 GPFSnode16 GPFSnode16

: : :

Add SAP HANA host names and IP addresses of all cluster nodes to the /etc/hosts file.

192.168.20.101 HANAnode01 HANAnode01 192.168.20.102 HANAnode02 HANAnode02

192.168.20.103 HANAnode03 HANAnode03

: : :

192.168.20.116 HANAnode16 HANAnode16 : : :

Add the SAP HANA external host names and IP addresses of all cluster nodes to the /etc/hosts file. This is the hostname which will be used for the installation of the SAP HANA database and the connection with tools such as the SAP HANA Studio.

SSH configuration

Configure SSH keys to enable ssh and scp for password-less GPFS and SAP HANA communication:

#ssh-keygen -t rsa -f ~/.ssh/id_rsa -N '' # ssh-copy-id -i ~/.ssh/id_rsa.pub root@GPFSnodeNN # ssh-copy-id -i ~/.ssh/id_rsa.pub root@HANAnodeNN


Note: NN is the node number within the cluster. In a clustered configuration exchange SSH keys with all nodes using a script such as the following:

#!/bin/bash

for host in `cat /etc/hosts | grep GPFSnode | awk '{print $3}'`; do ping -c 1 -s 1 $host if [ $? -eq 0 ]; then ssh-copy-id -i ~/.ssh/id_rsa.pub root@$host fi done for host in `cat /etc/hosts | grep HANAnode | awk '{print $3}'`; do ping -c 1 -s 1 $host if [ $? -eq 0 ]; then ssh-copy-id -i ~/.ssh/id_rsa.pub root@$host fi done

Install GPFS packages

Install GPFS as described in Concepts, Planning and Installation Guide, available here.

Make sure to select the correct version regarding your GPFS release. Log in as user root. Install GPFS (base package):

# cd /var/tmp/install/GPFS-<GPFS-RELEASE>

# rpm -ivh GPFS.base-<GPFS-RELEASE>-0.x86_64.rpm

Verify the GPFS version installed.

Update to the latest GPFS maintenance release

Install the following four packages for the latest (X) maintenance release:

# rpm -Uvh GPFS.base-<GPFS-RELEASE>.x86_64.update.rpm

http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.gpfs.doc/gpfsbooks.html


# rpm -Uvh GPFS.docs-<GPFS-RELEASE>.noarch.rpm # rpm -Uvh GPFS.gpl-<GPFS-RELEASE>.noarch.rpm

# rpm -Uvh GPFS.msg.en_US-<GPFS-RELEASE>.noarch.rpm

Build the GPFS portability layer

Follow the instructions in /usr/lpp/mmfs/src/README:

# cd /usr/lpp/mmfs/src # make Autoconfig # make World # make InstallImages

Prepare GPFS configuration 1) In a clustered configuration these commands must be executed on all nodes. Add the GPFS

executables to the path by creating a file /etc/profile.local with this content:

2) Change file permission: # chmod 644 /etc/profile.local

3) Activate the new PATH variable:

# source /etc/profile.local

4) Create a dump-directory for GPFS: #mkdir /tmp/GPFSdump

5) Create a configuration-directory for GPFS:

# mkdir /var/mmfs/config


GPFS cluster configuration 1) Create a GPFS disk configuration file in /var/mmfs/config/ directory for SAP HANA

persistent storage. In the following example it is named as disk.list.data2.

2) Have the disk paths populated in GPFS disk definitions file, like syntax shown below.

durxha238129:~ # cat /var/mmfs/config/disk.list.data2 %nsd: device=/dev/dm-10 Device Mapper file (Storage Volumes) nsd=MDdata012 usage=dataAndMetadata failureGroup=1001 %nsd: device=/dev/dm-11 nsd=MDdata022 usage=dataAndMetadata failureGroup=1001 %nsd: device=/dev/dm-12 nsd=MDdata032 usage=dataAndMetadata failureGroup=1001 %nsd: device=/dev/dm-13 nsd=MDdata042 usage=dataAndMetadata failureGroup=1001 Note: Device paths can be verified using multipath –ll command as shown below.


While testing the configuration, Lenovo and EMC used the best practices to configure EMC VMAX storage to achieve SAP KPI’s required in TDI scenarios. EMC configuration guides for VMAX and VNX can be downloaded from the following links:

• http://www.EMC.com/collateral/white-paper/h12342-storage-config-best-practices-SAP-HANA-tdi-VMAX-VMAX3-wp.pdf

• https://community.EMC.com/docs/DOC-36203

3) On the first node, create the node configuration file /var/mmfs/config/nodes.cluster and add a line containing the internal GPFSnodenn host name for each node in the cluster. Append ":quorum" to the first three entries.

Example for a 4-node cluster:

GPFSnode01:quorum GPFSnode02:quorum GPFSnode03:quorum GPFSnode04

4) Create a cluster. − Single node:

# mmcrcluster -n /var/mmfs/config/nodes.cluster -p GPFSnode01 -r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster –A

Cluster: #mmcrcluster -n /var/mmfs/config/nodes.cluster -p GPFSnode01 -s GPFSnode02 -r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster -A

Note: All cluster commands must be run on the first node of cluster.

Example:

5) Assign a GPFS server license to the node: # mmchlicense server --accept -N all

http://www.emc.com/collateral/white-paper/h12342-storage-config-best-practices-sap-hana-tdi-vmax-vmax3-wp.pdf


https://community.emc.com/docs/DOC-36203


6) Adjust various GPFS settings. Run the following command on the first node of the cluster.

Note: Enter 999 when asked for Enter due to unknown attribute specified: skipDioWriteLogWrites

Note: Enter 999 when asked for Enter due to unknown attribute specified: prefetchAggressivenessWrite

# mmchconfig dataStructureDump=/tmp/GPFSdump,pagepool=4G,maxMBpS=2048, maxFilesToCache=4000,skipDioWriteLogWrites=1,nsdInlineWriteMax=1M,prefetchAggressivenes sWrite=2,readReplicaPolicy=local,enableLinuxReplicatedAIO=yes,restripeOnDiskFailure=yes,nsd ThreadsPerDisk=24

7) Create GPFS disks.

On Node 1, run the following command with NN being the node number.

# mmcrnsd –F /var/mmfs/config/disk.list.data.GPFSnodeNN

Note: If the command fails because the disk is already part of a cluster, append “-v no” to the command to disable the check.

8) Create the GPFS file system. Run the following command on the first node of the cluster.

# mmcrfs sapmntdata -F /var/mmfs/config/disk.list.data2 -A yes -B 64K -N 3000000 -v no -m 1 –M 3 -r 1 -R 3 -Q yes -T /sapmnt --block-group-factor 1 –write-affinity-depth 1


Note: The file system mount point is /sapmnt and the associated device name is /dev/sapmntdata.

For the directories /sapmnt/data, /sapmnt/log and /sapmnt/shared, GPFS filesets will be used instead of separate mounted file systems.

9) Verify the GPFS file system attributes/parameter values. #mmlsfs all


10) Mount the sapmntdata file system.

# mmmount /sapmnt –a

11) Create filesets HANAdata, HANAlog and HANAshared. Link the filesets to the file system.

# mmcrfileset sapmntdata HANAdata -t "Data Volume for HANA database" # mmcrfileset sapmntdata HANAlog -t "Log Volume for HANA database" # mmcrfileset sapmntdata HANAshared -t "Shared Directory for HANA database"

# mmlinkfileset sapmntdata HANAdata -J /sapmnt/data # mmlinkfileset sapmntdata HANAlog -J /sapmnt/log # mmlinkfileset sapmntdata HANAshared -J /sapmnt/shared

12) Start GPFS on all nodes and check the status.

# mmstartup –a and # mmgetstate –a


XFS configuration for SAP HANA scale-out using storage connector API

Network configuration

Gather the network details listed in the following table before you configure XFS based HANA scale out cluster.

Table 8. XFS Network configuration requirements in a SAP HANA scale out deployment

Network Description IP Addresses Host Name Net Mask Gateway

SAP HANA Internodes Communication

192.168.20.X

HANAnode0X 255.255.255.0 None

IMM Customer Defined

Customer Defined Customer Defined Customer Defined

Customer Network Customer Defined

Customer Defined Customer Defined Customer Defined

SAP HANA inter-node communication private IPs to be configured on the 10G NIC ports. Example: In a 3 node (2*worker + 1*standby) cluster setup. You will have the following network configurations:

Node 1 192.168.20.1 (HANAnode01), Node 2 192.168.20.2 (HANAnode02), Node 3 192.168.20.3 (HANAnode03),

Note: It is mandatory to use 10G NIC to meet the SAP defined KPIs in a TDI network.

XFS file system configuration for 2+1 HANA scale out

XFS can be used as an underlying file system for both HANA Data and Log volumes. NFS file system is used to create a shared file system which is mandatory in an SAP HANA scale-out scenario.

SAP HANA storage connector API will act as an agent for managing the shared volumes from EMC storage array to Lenovo System x servers, which will manage automatic mounting/unmounting of devices on HANA nodes. SCSI-3 Persistent Reservations are written to the devices to prevent any data corruption due to multiple hosts accessing the same device at any point of time.

For storage configuration best practices for EMC storage in a SAP HANA environment, please refer to the following technical white papers:


• http://www.EMC.com/collateral/white-paper/h12342-storage-config-best-practices-SAP-HANA-tdi-VMAX-VMAX3-wp.pdf







1) Once the storage volumes are presented to the HANA Nodes, start the XFS file system creation on the block devices. Run the following command to discover the shared disks to be used for HANA Data and Log volumes for all the nodes in the scale-out.

Note: This test used a 3 node (2*worker+1*standby) scale out configuration.

# multipath –ll

2) Create the XFS file system using the command, #mkfs.xfs <block device>.

Example : #mkfs.xfs /dev/mapper/360000970000195702343533030344346

#mkfs.xfs /dev/mapper/360000970000195702343533030343737

3) Create directories for mounting the above file-systems.

#mkdir /hana/data

#mkdir /hana/log

#mkdir /hana/shared

4) Verify whether you can mount the file system on /hana/data and /hana/log mount points with the following options:

− Node 1 (HANA role: worker) log volume


#mount /dev/mapper/360000970000195702343533030344346 /hana/log –o rw,inode64, nobarrier

− Node 1(HANA role: worker) data volume

#mount /dev/mapper/360000970000195702343533030343737 /hana/data –o rw, inode64

− Similarly you need to verify the XFS file system creation on shared block devices from node 2 (HANA role: worker).

− Node 3 (HANA role: standby)

This will not have any XFS mount points as it is taken care of by SAP HANA storage connector API.

5) After verifying the XFS file system, unmount the file systems since this will be managed automatically by SAP HANA storage connector API.

6) Make sure all the three nodes have a NFS file system shared between them and mounted on /hana/shared, as shown below:


SAP HANA global.ini file

The storage connector API is controlled in the storage section of the SAP HANA global.ini file. This section contains entries for the block devices with optional mount options. The WWIDs of the partition entries can be determined using the multipath –ll command on the HANA hosts.

Below shows global.ini was used in the test.

1) Copy the global.ini file to /hana/shared directory, which is hosted on a NFS share and available to all the three nodes in scale out.

2) Make sure password less ssh works between all the three nodes. Refer to the SSH configuration” section for steps to configure the same.

HANA DB installation

1) Based on the SID to be selected for HANA installation, create a directory with SID name under /hana/data, /hana/log, This particular test has SID= ‘ANA’ and set the execute permission to directory.

#mkdir /hana/data/ANA

#mkdir /hana/log/ANA

# chmod 777 /hana/data/ANA

# chmod 777 /hana/log/ANA

2) Change directory to where the HANA bits are located.


3) Run the following command from the “master/worker node” to start multihost HANA DB server installation

#./hdblcm –action=install –storage_cfg=/hana/shared –addhosts=<worker node hostname>:role=worker,<standby node hostname>:role=standby

Note: This test uses SAP HANA SPS09 version.

4) The following details are required while HANA DB server installation.

− Which component needs to be installed? Select option ‘1’ for Server.

− Enter Local hostname: Usually its populated with the hostname, if not enter the master/worker node hostname

− Root password: <customer defined>


− Enter System Instance ID: SID (In this test we have used ‘ANA’)

− Enter Instance Number: By default, it will be listed based on the number of instances loaded on the server.


− Depending on the Single DB/Multi tenant DB configuration, you can opt for options. This test used ‘1’: Single Container

− Depending on the DB usage you can opt. This test used ‘4’’: Custom.

− Certificate Host Name can be accepted as in the options.

− DB admin user is created with name <sid>adm and password needs to be set. P lease refer to the HANA server installation guide for password rules.

− After the summary, HANA DB installation starts.

5) Follow the SAP HANA server installation guide for more details.

Troubleshooting tip: If there is an installation error due to directory access, then make sure there is “<sid>adm:SAPsys” permission assigned for the ‘ANA’ directory

#chown <sid>adm:SAPsys /hana/data/ANA

#chown <sid>adm:SAPsys /hana/data/ANA

http://help.sap.com/hana/SAP_HANA_Server_Installation_Guide_en.pdf


6) Once HANA is installed and SAP HANA storage connector API is active on the nodes, the mount points for SAP HANA data, log volumes would look like the following three Node 1, Node 2, and Node 3 examples:

Node 1 (Master/worker):


Node 2 (Worker):


Node 3 (Standby): No /hana/data and /hana/log will be mounted on a standby node prior to a failover. Should a failover occur, SAP HANA storage connector API will mount the respective failed node HANA partitions in standby node.

7) Run the following commands to check HANA services status.

− #su - <sid>adm

− #SAPcontrol –nr 00 –function GetSystemInstanceList

You should see the output as follows; color designates the status of HANA DB in each node.


HA tests (Scale-out node failover test) SAP HANA scale-out node failover test was performed on a SAP HANA storage connector API based configuration, as previously discussed.

Worker node failover This section describes how to check the failover capability of HANA. To check this, service must be stopped.

Run, # su - <sid>adm

# SAPcontrol –nr <instance_number> -function Stop HDB

Before HDB stop was issued, the following shows the status of scale-out configuration:

Post failover, the following shows that the standby node has taken over the role of worker.

/hana/data and /hana/log mount points are failed over from the failed worker node, to the standby node, as shown in the following:


Master/Worker node failover – Hardware failure simulation This test used the #echo b > /proc/sysrq-trigger command, which is a SIGKILL to all running process without any gracious file system unmount. After running the above command, the status of HANA services was checked, as shown in the following example.

Once the HANA services failed over to the standby node, the standby node inherits the role of the master/worker node.

Appendix A This section describes the Lenovo System x server and EMC VMAX storage array model details used in these tests.

HANA compute node details

• 3*Lenovo System x x3850 X5 Intel Xeon CPU E7- 8870 @ 2.40GHz, 512GB RAM

− QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02) − Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02) − SuSE Linux 11 SP3 for SAP application with SAN Boot (64GB LUN for root and 32GB

LUN for swap file system) − IBM GPFS 4.1.0.7 − DM-MPIO

• 3*Lenovo Flex System x880 x6 Intel Xeon E7-8880 v2 @ 2.5GHz, 256GB RAM

− Flex System FC-5022 16GB SAN scalable switch installed. − 2*onboard EN2024 4-port 1GB Ethernet adapter. − SuSE Linux 11 SP3 for SAP application with SAN Boot (64GB LUN for root and 32GB

LUN for swap file system)


− IBM GPFS 4.1.0.7 − DM-MPIO − Dual port 16GB QLogic HBAs, Model# QM12672-IBM-BK

SAP HANA database version used for the tests

• Version 1.00.60.379371

• 3-Node scale-out configuration (2 worker + 1 stand-by)

SAN details

• EMC VMAX 40K (5876.272.177) – 2 Engines (Using 6 FA processors) connected via Brocade SAN switches.

• Brocade DS-6510 16 GB port speed departmental switch.

Appendix-B

Lenovo compute building blocks for SAP HANA TDI

Table 9 below shows the Lenovo System-X server’s supported in a SAP HANA TDI environment

Table 9. Lenovo System x server’s and components certified for SAP HANA TDI

Machine Type CPU Type Memory FC HBA’s Ethernet cards

X3850 X6 (Type 6241)

Intel Ivy Bridge EX E7 8880v2,

2S and 4S

Maximum 6 TB1 Click Here Click Here

X3950 X6 (Type 6241)

Intel Ivy Bridge EX E7 8880v2,

4S and 8S

Maximum 12 TB 2

Click Here Click Here

Flex System x240 (Type

7863)

Intel Xeon E5-26xx v2, 2S

Maximum 768 GB


Flex System x240 (Type

8737)


Maximum 768 GB


Lenovo nx360 M5 (Type 5465)


Maximum 512 GB


1 6 TB for Single Node SoH implementations only, 8S 2 TB is maximum BW scale out supported configuration in TDI. For more information please check SCN.

2 12 TB support only for non-production environment.

http://www.redbooks.ibm.com/technotes/tips1084.pdf


http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/tips1132.html#hba

http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/tips1132.html#hba

http://public.dhe.ibm.com/common/ssi/ecm/wz/en/wzb12345usen/WZB12345USEN.PDF




http://www.redbooks.ibm.com/abstracts/tips1195.html


http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/appliances.html#categories=Lenovo


Lenovo System x 3500 M4 Type 7383


Maximum 384 GB




Maximum 768 GB




Maximum 768 GB


BladeCenter System x HS23

Type 7875


Maximum 512 GB


Lenovo x240 M5 Compute Node

(9532)


Maximum 1.5 TB


Lenovo x3500 M5 (5464)


Maximum 1.5 TB


Lenovo x3550 M5 (5463)


Maximum 1.5 TB


Lenovo x3650 M5 (5462)


Maximum 1.5 TB


Flex System X880 X6

Intel Xeon E5-26xx v2, 2S,4S

and 8S

Maximum 6 TB


General capacity considerations for VMAX storage in TDI

In general, the required capacity for the SAP HANA persistence on disk depends on the in-memory database size and the RAM size of the HANA servers.

Every SAP HANA customer must perform memory and CPU sizing as the first step to sizing a SAP HANA deployment. For new SAP HANA implementations, size the memory and CPU for a SAP HANA system using the HANA version of the SAP Quick Sizer tool, available on the SAP Service Marketplace website or consult SAP for assistance. For systems that are migrating to SAP HANA, SAP provides tools and reports for proper HANA memory sizing.

After you determine the memory requirements, you can estimate the disk capacity requirements by using the sizing rules in the SAP white paper, SAP HANA Storage Requirements. File systems must include:

• 1x RAM for the data file system

• ½x RAM for the log file system with 512 GB or less, or at least 512 GB for the log file system with 512 GB RAM or higher



http://www.redbooks.ibm.com/abstracts/tips0851.html#contents


http://www.redbooks.ibm.com/abstracts/tips0850.html?Open#contents














https://hcp.sap.com/content/dam/website/saphana/en_us/Technology%20Documents/Storage%20Whitepaper%202%204.pdf

https://hcp.sap.com/content/dam/website/saphana/en_us/Technology%20Documents/Storage%20Whitepaper%202%204.pdf


SAP refers to RAM size as the size of the database in contrast to the physical memory size of the servers. For example, the HANA database can consume 1.3 TB RAM on a single host but the host has 2 TB physical RAM capacity. SAP recommends the sizing based of the actual database size, in this example 1.3 TB.

Note: SAP sizing requirements do not consider future growth of the database.

However, in certain situations, you may need to expand the size of a data or log file system. Lenovo and EMC recommend sizing the database persistence (the data and log file systems) at a minimum on the physical RAM size of the HANA hosts.

To calculate the required usable storage capacity for the HANA persistence of a scale-up (single-host) or scale-out (multi-host) appliance, the following details are required:

• (A)—RAM size of a HANA worker host

• (B) —Number of HANA worker hosts

For example, use the following formulas to calculate the required capacity for a 6+1 HANA scale-out appliance where each server has 2 TB RAM:

• Total capacity for data = (A) * (B) = 2 TB * 6 = 12 TB

• Total capacity for Log = 512 GB * (B) = 512 GB * 6 = 3 TB

• Total usable capacity for the HANA persistence = 12 TB + 3 TB = 15 TB

For more information on disk considerations, etc, please refer to VMAX best practices guide for HANA TDI

QLogic Fibre Channel Adapters for Lenovo System-X Servers

The QLogic 16 Gb FC Single-port and Dual-port Host Bus Adapters (HBAs) for Lenovo System offer 16 Gbps line-rate performance at extremely low CPU utilization with full hardware offloads.

The Flex System FC5172 2-port 16 Gb FC Adapter enables high-speed access for Flex System compute nodes to connect to a Fibre Channel storage area network (SAN). This adapter is based on the proven QLogic 16 Gb ASIC design and works with the 8 Gb and 16 Gb Flex System Fibre Channel switches and pass-through modules.

Management software

• The QLogic QConvergeConsole (QCC) management software delivers a unified web-based single-pane-of-glass management console across the QLogic family of storage and networking adapters. A graphical user interface (GUI) or command-line interface (CLI) are available. A VMware vCenter plug-in is also available.

• Common IT tasks, such as VLAN configuration and teaming, can be easily accomplished either through the QConvergeConsole or through native OS tools, minimizing IT training and deployment costs.




• Role-based authentication allows for separate logins and access for SAN and LAN administrators. This eliminates the need to change your organizational structure as you converge your network.

Figure 4. Figure 3: QLogic Single and dual port adapter

The above figure shows QLogic adapters used in the above SAP HANA TDI tests. These support nearly twice the throughput and 2.5 times the I/O operations per second (IOPS) per port compared to 8 Gb adapters. The QLogic 16 Gb HBAs are ideal for high bandwidth and I/O-intensive applications, such as SAP HANA. Through rigorous testing carried out using the ServerProven program, you can maintain a high degree of confidence that your storage subsystem is compatible and functions reliably when using these 16 Gb HBA adapters. Refer here for more information on QLogic Fibre channel adapter details supported for Lenovo System x servers.

HWCCT

The SAP HANA HW Configuration Check Tool (HWCCT) allows you to check the interoperability of SAP HANA with your existing enterprise storage in production environments. The SAP HANA HW Configuration Check Tool is a framework that provides tests and reports for new single host and scale out systems to determine if the hardware you intend to use meets the minimum performance criteria required to run SAP HANA in production use. Click Here for more details on HWCCT.

Resources EMC documentation:

• EMC VNX best practices for HANA TDI

• EMC-Lenovo white paper on HANA TDI

• EMC SRDF White paper

• EMC technical reference guides for SAP HANA TDI

http://www.qlogic.com/OEMPartnerships/IBM/Documents/adapters_IBM_servers.pdf

http://help.sap.com/saphelp_hanaplatform/helpdata/en/44/226b888ae64f40a6e7906eeaa0a44c/content.htm?frameset=/en/65/af0e2d971b430fb58b29323e2e1592/frameset.htm&current_toc=/en/00/0ca1e3486640ef8b884cdf1a050fbb/plain.htm&node_id=136






Lenovo documentation:

• Lenovo System x solution for SAP HANA

• Flex System X6 Solution for SAP HANA

• System x solutions for SAP environments

• In Memory computing on Lenovo eX5 and X6

• Lenovo System x solution for SAP HANA and VMware vSphere on X6

SAP documentation:

• SAP HANA Server Installation Guide

• Best Practice Document – Technical Deployment Options for SAP Systems with SAP HANA

• White Paper - SAP HANA System Landscape Guide

• SAP HANA Appliance

• SAP HANA One

• SAP HANA Enterprise Cloud

• SAP HANA tailored data center integration (integration of enterprise storage systems with SAP HANA)

• SAP HANA High Availability Paper

• SAP SCN page for TDI certified server’s and storage

QLogic documentation:

• Qlogic products for Lenovo System x

http://www-03.ibm.com/systems/x/solutions/sap/hana/

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=SP&infotype=PM&appname=LNVE_LZ_WZ_USEN&htmlfid=LZS12346usen&attachment=LZS12346USEN.PDF#loaded

http://www-03.ibm.com/systems/x/solutions/sap/

http://www.redbooks.ibm.com/abstracts/sg248086.html?Open

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=SP&infotype=PM&appname=STGE_XS_XG_USEN&htmlfid=XSS03127USEN#loaded

http://help.sap.com/hana/SAP_HANA_Server_Installation_Guide_en.pdf

https://proddps.hana.ondemand.com/dps/d/preview/6932132d32b54f58a961a755e043e21e/1.0/en-US/index.html#url=dfeb8da5f24749d089917673fe4ed629.html

https://proddps.hana.ondemand.com/dps/d/preview/6932132d32b54f58a961a755e043e21e/1.0/en-US/index.html#url=dfeb8da5f24749d089917673fe4ed629.html

http://scn.sap.com/docs/DOC-60282

http://help.sap.com/hana_appliance

http://help.sap.com/hana_appliance

http://help.sap.com/hana_one

http://hana.sap.com/platform/sap-hana-enterprise-cloud.html

http://hana.sap.com/abouthana/hana-technology/data-center.html

http://hana.sap.com/abouthana/hana-technology/data-center.html

http://www.saphana.com/docs/DOC-2775

http://global.sap.com/community/ebook/2014-09-02-hana-hardware/enEN/appliances.html#categories=Lenovo

http://www.qlogic.com/OEMPartnerships/ibm/Pages/IBMOEMProducts.aspx?productFamily=adapters&productName=FibreChannelAdapters

SAP HANA TDI Configuration Guidelines for Lenovo System x ...

Documents

Transcript of SAP HANA TDI Configuration Guidelines for Lenovo System x ...