Network Design and Configuration Guide -...
Transcript of Network Design and Configuration Guide -...
FlexFrame® Orchestrator
Version 1.1A
Network Design and Configuration Guide
Edition March 2015 Document Version 1.1
Fujitsu Limited
© Copyright Fujitsu Technology Solutions 2014
FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Limited in
Japan and other countries.
SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany
and in several other countries
Linux® is a registered trademark of Linus Torvalds
SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other coun-
tries
Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates
Intel® and PXE® are registered trademarks of Intel Corporation in the United States and oth-
er countries
MaxDB® is a registered trademark of MySQL AB, Sweden
MySQL® is a registered trademark of MySQL AB, Sweden
NetApp® and the Network Appliance® logo are registered trademarks and Network Appli-
ance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other countries.
VMware®, ESX®, ESXi, VMware vCenter, VMware vSphere are registered trademarks or
trademarks of VMware, Inc. in the United States and/or other jurisdictions.
Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel
Corporation
Windows® and Word® are registered trademarks of Microsoft Corporation
All other hardware and software names used are trademarks of their respective companies.
All rights, including rights of translation, reproduction by printing, copying or similar methods,
in part or in whole, are reserved.
Offenders will be liable for damages.
All rights, including rights created by patent grant or registration of a utility model or design,
are reserved.
Delivery subject to availability. Right of technical modification reserved.
Network Design and Configuration Guide
Contents
1 Introduction ..................................................................................................... 1 1.1 Purpose of this Document ................................................................................. 1 1.2 Notational Conventions ..................................................................................... 1 1.3 Document History .............................................................................................. 1 1.4 Related Documents ........................................................................................... 2
2 Concept and Design ........................................................................................ 3 2.1 Switch Groups ................................................................................................... 3 2.2 Network Speed .................................................................................................. 5 2.3 Jumbo Frames .................................................................................................. 5 2.4 Network Connection .......................................................................................... 6 2.5 Linux Bonding .................................................................................................... 7 2.6 ESX NIC Teaming ............................................................................................. 8 2.7 NetApps Interface Grouping .............................................................................. 9 2.8 BX Link State Propagation ................................................................................ 9 2.9 Virtual LAN ...................................................................................................... 10 2.10 Uplink Connection ........................................................................................... 13 2.11 Client LAN Connection .................................................................................... 13
3 FlexFrame Network Versions ....................................................................... 15 3.1 Small Version .................................................................................................. 15 3.2 Directly Connected Versions ........................................................................... 15 3.3 Enterprise Version ........................................................................................... 19
4 Configuration Commands ............................................................................ 21 4.1 Linux Bonding Interface ................................................................................... 21 4.2 Linux VLAN Interface ...................................................................................... 22 4.3 NetApp Filer Configuration .............................................................................. 22 4.4 Network Switch Configuration ......................................................................... 23
5 API - Network ................................................................................................. 28 5.1 XML Description .............................................................................................. 28 5.2 HW Quickguide ............................................................................................... 30 5.3 Perl Module ..................................................................................................... 31
6 Abbreviations ................................................................................................ 35
7 Glossary ......................................................................................................... 37
8 Index ............................................................................................................... 40
Network Design and Configuration Guide 1
1 Introduction
1.1 Purpose of this Document
This document describes the FlexFrame networking concept and design and possible
networking topologies.
The reader should be familiar with IP networking and needs basic knowledge about
Ethernet-based networking with switches and virtual LANs. Knowledge of network switch
configuration is recommended. The knowledge of the FlexFrame Installation Guide and
the FlexFrame Administration and Operation Guide would be an advantage.
1.2 Notational Conventions
The following conventions are used in this manual:
Additional information that should be observed.
Warning that must be observed.
fixed font Names of paths, files, commands, and system output.
<fixed font> Names of variables.
fixed font User input in command examples
(if applicable using <> with variables).
1.3 Document History
Document Version Changes Date
1.0 First Edition 2014-12-03
1.1 Related Documents modi-
fied
2015-03-25
Introduction Related Documents
2 Network Design and Configuration Guide
1.4 Related Documents
FlexFrame® – Administration and Operation
FlexFrame® – HW Characteristics Quickguides
FlexFrame® – Installation and Configuration of LVM 2.1 Standard Edition
FlexFrame® – Installation Guide for SAP Solutions
FlexFrame® – Installation of a FlexFrame Environment
FlexFrame® – Management Tool
FlexFrame® – FlexFrame Agents Installation and Administration
FlexFrame® – FlexFrame Messenger Installation and Administration
FlexFrame® – FlexFrame LogAgent Installation and Administration
FlexFrame® – Security Guide
FlexFrame® – Technical White Paper
FlexFrame® – Upgrading FlexFrame Orchestrator 1.0A to 1.1A
ServerView Documentation
SUSE Linux Enterprise Server Documentation
Network Design and Configuration Guide 3
2 Concept and Design
The design goals for the FlexFrame network concept:
avoid single points of failure (SPOF)
avoid unnecessary complexity
are achieved by the following means:
use redundant network
use a lightweight design
use standard interfaces and configurations
keep interface configuration as identical as possible on all server systems
keep design flexible enough to meet the requirements from small to medium up
to enterprise sized installations
reduce count of needed interfaces per system
and results in the concept described below.
2.1 Switch Groups
Switch 2
Switch 1
SwitchGroup
LAN A
LAN B
Application Node
PORT a
Application Node
Blade Rack
PORT c PORT b
Control Node 2
PORT a
PORT c PORT b
LAN A
LAN B
Application Node
LAN A
LAN B
Control Node 1
LAN A
LAN B
NAS
LAN A
LAN B
The very simplified drawing above may visualize the general concept (only data connec-
tions are considered). The core of the design is the switch group which provides redun-
dant usable ports. If one member of the switch group fails the ports of the other members
Concept and Design
4 Network Design and Configuration Guide
are still working. Every end system and the uplink are using this redundancy as described
later. In this way a failure of a single member of the switch group is tolerated and allows
continuous operation.
FlexFrame distinguishes the following switch group types:
CAT3750-STACK
Two up to nine switches of the Cisco Catalyst 3750G, 3750E or 3750X switch family
are building a Cisco Catalyst switch stack. The switches are connected via Cisco
StackWise cabling and behave like a single switch. For more than 4 switches with
10GbE ports the StackWise cabling may be a bottleneck.
ICX6430-STACK
Two up to four switches of the Brocade ICX 6430 switch family are building a Bro-
cade FastIron traditional switch stack. The switches stacking ports are connected via
standard ethernet cabling and the switch stack behave like a single switch.
ICX6450-STACK
Two up to eight switches of the Brocade ICX 6450 switch family are building a Bro-
cade FastIron traditional switch stack. The switches stacking ports are connected via
standard ethernet cabling and the switch stack behave like a single switch.
NEXUS5000-VPC
Exactly two switches (nexus50xx) of the Cisco Nexus 5000 Switch family are building
a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel
between the switches FlexFrame configures two ports on each switch. For the vPC
peer-keepalive link as a necessary alternative Layer 3 connectivity between the
switches FlexFrame uses the mgmt0 interface IP addresses as recommended from
Cisco.
NEXUS5500-VPC
Exactly two switches (nexus55xx) of the Cisco Nexus 5000 Switch family are building
a Cisco Nexus vPC domain. For the vPC peer-link as a necessary special channel
between the switches FlexFrame configures two ports on each switch. For the vPC
peer-keepalive link as a necessary alternative Layer 3 connectivity between the
switches FlexFrame uses the mgmt0 interface IP addresses as recommended from
Cisco.
VCS-LC
Exactly two Brocade VDX 6740 switches are building a Brocade VCS Logical Chas-
sis. For the fabric inter switch link as a necessary special connection between the
switches FlexFrame configures two ports on each switch. In addition Out-of-Band
management is used as basis for the logical chassis management.
For further details about supported switches see the FlexFrame Support Matrix.
For further details about the switches see the Cisco manuals.
For further details about the purpose of the different switch group types see below.
Concept and Design
Network Design and Configuration Guide 5
2.2 Network Speed
FlexFrame supports network connections for data communication with the following net-
work speeds:
1Gbit/sec (1GbE)
10Gbit/sec (10GbE)
Pay attention to the fact that 10GbE connected servers may also request a 10GbE con-
nected NAS system and 10GbE uplinks between.
For further details about supported end systems see the FlexFrame Support Matrix.
2.3 Jumbo Frames
IEEE 802.3 defines the maximal payload size of Ethernet frames to 1500 bytes. This re-
sults in using a maximal transfer unit size of 1500 bytes on network interfaces (MTU size
1500).
NetApp recommends the use of MTU size 9000 (jumbo frames) for filer communication.
MTU size 9000 is standard with cDOT.
FlexFrame allows the use of MTU size of 9000 in storage networks and uses MTU size
1500 in all other networks. Switches are always configured to be able to transmit frames
with a payload of 9000 bytes.
Concept and Design
6 Network Design and Configuration Guide
2.4 Network Connection
Network connection is established by connecting ports of end systems to ports of network
devices which itself are connected to other network devices. The characteristics of con-
nected ports must be compatible and a proper connector must be used.
FlexFrame classifies the ports of network devices as follows:
classifier meaning supported from
TX 10/100/1000 ethernet port
RJ45 connector
Cisco Catalyst 3750 family
Brocade ICX 6430, ICX6450
FX 1GbE ethernet port
SFP connector
Cisco Catalyst 3750G
Cisco Nexus 5000 family
Brocade ICX 6430
Brocade VDX 6740
TGBx 10GbE ethernet port
SFP+ connector
Cisco Nexus 5000 family
Cisco Catalyst 3750E
Brocade VDX 6740
TX ports are requested from end systems for RJ45 ports for management or 1GbE data
connection and for uplinks using this uplink port media. Use Cat5 cable for connections
up to 100 Mbit/sec and cat5e cable for connections up to 1000 Mbit/sec.
FX ports are requested for uplinks using this uplink port media. Use LC-LC fiber optic ca-
ble to connect SFP connectors or 1GBASE-CU SFP cable (Twinax) to connect ports suit-
ed for SFP connectors.
TGBx ports are requested from end systems for 10GbE data connection and 10GbE up-
links. Use LC-LC fiber optic cable to connect SFP+ connectors or 10GBASE-CU SFP ca-
ble (Twinax) to connect ports suited for SFP+ connectors. Use SC-LC fibre optic cable to
connect a Cisco Catalyst 3750E 10GbE port with another SFP+ connector.
Consider the support matrix of devices with respect to the supported SFP/SFP+ connect-
ors.
Concept and Design
Network Design and Configuration Guide 7
2.5 Linux Bonding
Linux provides a mechanism called bonding which can aggregate several physical inter-
faces to a virtual interface (bond). A bond interface can be used like a usual interface and
is operational as long as at least one physical interface assigned to the bond is opera-
tional. Several bonding configurations are available.
Switch 2
Switch 1
active eth0
eth1
rack server or
server blade
bond0
active
When running Linux on a physical server (rack server or server blade) used as Applica-
tion Node FlexFrame aggregates 2 physical interfaces using adaptive load balancing
(mode 6) and failure detection relies solely on the link state provided by the adapter (MII
monitoring). This is an active/active method and has no requirements on the link partners
which can be switch blades or switch group switches. The requirement on the adapter to
be able to change the MAC address is fulfilled from all supported servers.
On the Control Nodes bonding mode 1 (active-backup) is used. No requirements have to
be fulfilled neither from the link partners nor from the adapter. Failure detection relies
solely on the link state provided by the adapter (MII monitoring).
Concept and Design
8 Network Design and Configuration Guide
2.6 ESX NIC Teaming
ESX provides a mechanism called NIC teaming which is used if multiple physical Ether-
net adapters are assigned to a single virtual switch (vSwitch). A vSwitch is operational as
long as at least one physical Ethernet adapter assigned to the vSwitch is operational.
Several NIC teaming policies are available and can be set on the port group level.
Switch 2
Switch 1
active vmnic0
vmnic1
rack server or
server blade
vSwitch active
FlexFrame assigns 2 physical Ethernet adapters in ESX default configuration to the
vSwitch and puts all these adapters in the active list for the FlexFrame generated port
groups. The adapters are used with load balancing based on the originating virtual Port
ID and failure detection relies solely on the link state provided by the adapter
This method has no requirements on the link partners which can be switch blades or
switch group switches.
Concept and Design
Network Design and Configuration Guide 9
2.7 NetApps Interface Grouping
NetApps Data ONTAP provides a mechanism called interface grouping which can aggre-
gate several physical interfaces to a virtual interface. An interface group can be used like
a usual interface and is operational as long as at least one physical interface assigned to
the group is operational. Several modes are available.
Switch 2
Switch 1
SwitchGroup
NetApp Filer
ifgrp
e0a
e0b
active
active
FlexFrame aggregates at least 2 physical interfaces into a dynamic multimode interface
group which is compliant with IEEE 802.3ad. All physical interfaces are active. The corre-
sponding ports on the switch group switches must build a channel also complaint with
IEEE 802.3ad (LACP).
2.8 BX Link State Propagation
BX switch blades provide a mechanism which allows associating several downstream
ports with an upstream port or channel for link state propagation. If the link of the up-
stream port or channel goes down the link of all associated downstream ports are also
forced to go down. In this way a bonding driver on a server blade will get aware of the
upstream problem and switches the traffic of concerned interfaces.
Concept and Design
10 Network Design and Configuration Guide
active
eth0
eth1
server blade
bond0 active
Switch 2
Switch 1
SwitchGroup BX chassis
Switchblade 1
Switchblade 2
FlexFrame associates all downstream ports of a switch blade and a single upstream
channel for link state propagation. The upstream channel is compliant with IEEE 802.3ad.
The corresponding ports on the switch group switches must also build a channel com-
plaint with IEEE 802.3ad (LACP). The channel consists of at least two ports with crosso-
ver cabling. More ports can be used for higher bandwidth.
2.9 Virtual LAN
Virtual LAN (VLAN) is a mechanism which allows having multiple separated networks on
the same physical network infrastructure. A widely used standard is tagged VLAN accord-
ing IEEE 802.1q. The protocol specifies an extension of the Ethernet header by a VLAN
identifier (tag). A packet belongs to the VLAN identified by the tag. Communication be-
tween VLANs is only possible by routing as is common for usual networks. For compati-
bility reason packets without a tag are also allowed. A switchport can be configured to
assign received packets without tag to a specific VLAN and remove the tag when a pack-
et of this VLAN should be send. The VLAN is then called native or untagged. End sys-
tems which access only one VLAN or cannot handle VLAN tags are connected this way.
End systems used within FlexFrame like NAS systems or servers running Linux are able
to handle VLAN traffic according IEEE 802.1q.
FlexFrame can handle different pools with independent networks. The traffic between
communication partner types (client – server, server – server and server – storage) is al-
so separated. Therefore VLANs are used as helpful mechanism to reduce the necessary
physical infrastructure and increase the flexibility with respect to the pooling concept.
According IEEE 802.1q the VLAN tag has a value range from 1 to 4094 but there are
switch specific restrictions.
Concept and Design
Network Design and Configuration Guide 11
Cisco Catalyst 3750 Switch Family
1005 VLANs are supported. Normal usable VLANs are in the range from 1 to 1001. VLAN
numbers 1002 through 1005 are reserved for Token Ring and FDDI VLANs. VLAN num-
bers 1006 through 4094 are extended-range VLANs and usable if not already used inter-
nally for routed ports.
Cisco Nexus 5000 Switch Family
507 VLANs are supported. Normal usable VLANs are in the range from 1 to 1005. VLAN numbers 1006 through 4094 are extended-range VLANs and usable with the following restrictions. The state is always active and VLANs 3968 to 4047 and 4094 are reserved for internal use.
Brocade ICX 6400 Switch Family
32 VLANs are supported. Normal usable VLANs are in the range from 1 to 4094. VLAN
IDs 4087, 4090, and 4093 are reserved for Brocade internal use only. VLAN 4094 is re-
served for use by Single STP.
Brocade VDX 6700 Switch Family
507 VLANs are supported. Normal usable VLANs are in the range from 1 to 3959. VLAN IDs 3960 through 4090 are internally-reserved VLAN IDs. However, the reservedvlan command can modify this range. VLANs above 4090 are not configurable. VLAN 1002 is the default FCoE VLAN.
Concept and Design
12 Network Design and Configuration Guide
Physical Network Layout & Virtual LANs per Pool
V4: Client VLAN
V3: Control VLAN
V2: Server VLAN
V1: Storage VLAN
V1
V1 V2 V3 V4
Bond 0
eth 2 eth 1eth 0
IPMI
Onboard
LAN Port 3
Onboard
LAN Port 1
V1 V2 V3 V4
Bond 0
eth 2 eth 1eth 0
IPMIV1 V2 V4
Bond 0
eth 2 eth 1eth 0
IPMI
V3
ifgrp a1a
Physical View:
Redundant Connections
Control Node 1 Control Node 2 Application Node 1
Application Node n
Logical View: VLANs
Onboard
LAN Port 2
Onboard
LAN Port 3
Onboard
LAN Port 1
Onboard
LAN Port 2eXa eXb
Onboard
LAN Port 3 LAN Port 1 LAN Port 2
Switch 2Switch 1
NAS
System
Control Nodes and Linux Application Nodes (RX300), NetApp NAS (7-mode)
Switchgroup
The picture shows the physical and logical view of networking in FlexFrame with respect
to a single pool. That is:
An Application Node is assigned to a pool and connected to the pools client, server and
storage VLAN. When running on ESX server for every connection an interface is created
on the vSwitch, when running native the interfaces for client and server LAN are created
as VLAN interface on top of the bond. The storage LAN is also used for PXE boot and
assigned native because no tags can be handled in this early phase. Therefore the stor-
age LAN is accessible via the bond directly.
An ESX server is connected to all VLANs necessary for the Application Nodes and to the
control LAN which is assigned native.
IPMI interfaces of rack servers and management blades of BX cabinets are connected
native to the control LAN.
A Control Node is connected to all VLANs, the control LAN is native.
A NAS system is connected to the control LAN and storage VLANs of pools the NAS sys-
tem supports, all VLANs are tagged.
Concept and Design
Network Design and Configuration Guide 13
2.10 Uplink Connection
Switch 2
Switch 1
SwitchGroup
FlexFrame can assign uplinks to a switch group. An uplink consists of ports from different
switch group members building a channel and carries the traffic of all used VLANs. An
uplink is used to connect a switch group to another switch group or to the customer cor-
porate LAN.
Examples for supported topologies are given in chapter 3.
When connected to customer corporate LAN the connected customer ports must also
build a channel. The channel must be compliant with IEEE 802.3ad (LACP) and all used
VLANs must be allowed tagged.
2.11 Client LAN Connection
For accessing a FlexFrame infrastructure solution connection to the client LAN must be
established. In general this means the client LAN must be available in the context of a
router and appropriate routing must be configured. This is a manual task and the custom-
er decides which customer network should have access to a special client LAN.
To bring a client LAN in the context of a customer router different ways are available:
a) If uplinks of switch groups are connected to the customer corporate LAN e.g. as
described in chapter 3.3 the client LANs are already available and no further ar-
rangements are necessary.
Concept and Design
14 Network Design and Configuration Guide
b) Special uplinks for client LAN connections are configured (e.g. see
ff_swgroup_adm.pl) and connected to the customer corporate LAN like normal
uplinks. The VLANs on these uplinks are restricted to client LAN VLANs.
c) Special ports for client LAN connections are configured. These CLAN ports are
thought being connected to a router and therefore configured as spanning tree
edge ports. The ports can be distributed over several switch groups and it is
possible to have particular ports for every client LAN. Only Ports of switches of a
CAT3750-STACK are allowed to be used as CLAN ports. See below an exam-
ple how CLAN ports configured for a single client LAN can be used.
The ports configured for client LAN connection are connected to Layer 3 ports of different
Layer 3 switches. An IP address is assigned to every Layer 3 port and the Layer 3
switches are configured to build a logical router e.g. via HSRP. The logical router sup-
ports a virtual IP address (IP3) which is used as standard gateway for the special client
LAN.
Switch 2
Switch 1
SwitchGroup 1
Switch 2
Switch 1
SwitchGroup 2
uplink channel
Virtual Gateway Adress IP3
Layer 3 port IP1
Layer 3 port IP2
outside FlexFrame
Network Design and Configuration Guide 15
3 FlexFrame Network Versions
The FlexFrame network design is flexible enough to meet requirements from small to en-
terprise size networks. In the following some examples are given.
3.1 Small Version
This version is based on one switch group as there is no need for
splitting FlexFrame into different data centers
providing more switch ports as one switch group can offer.
Switch 2
Switch 1
CAT3750-STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
client LAN
All devices are connected to a single CAT3750-STACK. The connection of the devices is
shown simplified only with regard to the redundant connections for data communication.
For client LAN connection two ports are configured on different members of the switch
group. No uplink needs to be configured.
3.2 Directly Connected Versions
Topologies according to the rules mentioned in 2.10 with more than one switch group and
without any uplink connection to the customer corporate LAN are called directly connect-
ed versions.
The following version may meet the requirements if
FlexFrame has to be split into two data centers or
one switch group might have not enough ports for all devices
FlexFrame Network Versions
16 Network Design and Configuration Guide
Example1: Two CAT3750-STACKs
Switch 2
Switch 1
CAT3750- STACK
Switch 2
Switch 1
CAT3750- STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
Uplink Channel
client LAN
All devices are connected to one of two CAT3750-STACKs. The connection of the devic-
es is shown simplified only with regard to the redundant connections for data communica-
tion. For client LAN connection one port is configured on every switch group.
The switch groups are connected via uplink channel a) directly without any other network Layer2 device between per copper or for
greater distance per fiber optic cable.
b) via network devices allowing a Layer2 connection e.g. DWDM for even greater
distance. In this case the network devices between must be transparent to the
switch groups, especially regarding link state. That means, if a port is shutdown
on one switch group the corresponding port on the other switch group must see
a link down event. Also if the connection between is broken both ports must see
a link down event.
FlexFrame Network Versions
Network Design and Configuration Guide 17
Example 2: single NEXUS5000-VPC with one CAT3750-STACK
Switch 2
Switch 1
NEXUS5000-VPC
Switch 2
Switch 1
CAT3750- STACK
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
Uplink Channel
client LAN
The CAT3750-STACK is directly connected to the NEXUS5000-VPC without any other
network Layer2 device between. The service LAN ports of rack servers and the control
nodes and the management ports of the nexus switches are connected to the CAT3750-
STACK. The end systems connected to NEXUS5000-VPC are 10GbE-connected. For
client LAN connection two ports are configured on different members of the CAT3750-
STACK.
FlexFrame Network Versions
18 Network Design and Configuration Guide
Example 3: two NEXUS5000-VPCs each with connection to one CAT3750-STACK
Switch 2
Switch 1
CAT3750-STACK
Switch 2
Switch 1
CAT3750-STACK
Switch 2
Switch 1
NEXUS5000-VPC
Switch 2
Switch 1
NEXUS5000-VPC LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
LAN A
LAN B Node
uplink channel
client LAN
The service LAN ports of rack servers and the control nodes are connected to a
CAT3750-STACK which is connected to the NEXUS5000-VPC. The end systems con-
nected to NEXUS5000-VPC are 10GbE-connected.
The NEXUS5000-VPC switch groups are connected via uplink channel
directly without any other network Layer2 device between per fiber optic cable.
via network devices allowing a Layer2 connection e.g. DWDM for greater dis-
tance. In this case the network devices between must be transparent to the
switch groups, especially regarding link state. That means, if a port is shutdown
on one NEXUS5000-VPC switch group the corresponding port on the other
NEXUS5000-VPC switch group must see a link down event. Also if the connec-
tion between is broken both ports must see a link down event.
FlexFrame Network Versions
Network Design and Configuration Guide 19
3.3 Enterprise Version
The possibilities of directly connected versions are restricted. Greater configurations need
uplinks to customer corporate LAN e.g. the customer core switches.
Customer own Core
Switch
Customer own Core
Switch
Customer Corporate LAN
Node
Node
Node
Node
Node
Node
Node
CAT3750-STACK
Uplink Channel
NEXUS5000-VPC
NEXUS5000-VPC
Node
Uplink Channel Uplink Channel
CAT3750-STACK
CAT3750-STACK
Uplink Channel
Uplink Channel
Each NEXUS5000-VPC switch group is connected with its uplink ports to core switch
ports. CAT3750-STACK switch groups may be connected to core switches or to
NEXUS5000-VPC switch groups. The switch groups may be connected to different core
switches. Like the Uplink the corresponding ports on the core switch(es) must:
build a channel
carry all VLANs used within FlexFrame
transfer traffic without blocking
The core switches are not part of FlexFrame, neither of automated configuration nor sup-
port. Configuration changes at core switches have to be done manually. There are no
messages or instructions from FlexFrame to support this.
Network Design and Configuration Guide 21
4 Configuration Commands
As described in Chapter 2 FlexFrame uses high availability configurations concerning
network on many devices. This chapter describes how this is done in detail.
4.1 Linux Bonding Interface
On the control nodes the bonding configuration is performed via configuration files
/etc/sysconfig/network/ifcfg-bond0:
STARTMODE=onboot
BOOTPROTO=static
IPADDR=192.168.20.1
NETMASK=255.255.255.0
NETWORK=192.168.20.0
BROADCAST=192.168.20.255
BONDING_MASTER=yes
BONDING_SLAVE_0=eth0
BONDING_SLAVE_1=eth1
BONDING_MODULE_OPTS="miimon=100 mode=6"
/etc/sysconfig/network/ifcfg-eth0
/etc/sysconfig/network/ifcfg-eth1 (both are the same):
STARTMODE=auto
On application nodes running native the bonding configuration is performed during boot
via commands. The network parameters are obtained through DHCP and include the IP
address for the storage LAN.
Configuration Commands
22 Network Design and Configuration Guide
4.2 Linux VLAN Interface
VLAN configuration is performed via configuration files
/etc/sysconfig/network/ifcfg-vlan<VLAN ID>
e.g. for VLAN 10: /etc/sysconfig/network/ifcfg-vlan10:
STARTMODE='onboot'
BOOTPROTO='static'
IPADDR=192.168.200.11
NETMASK=255.255.255.0
NETWORK=192.168.200.0
BROADCAST=192.168.200.255
ETHERDEVICE='bond0'
WIRELESS='no'
4.3 NetApp Filer Configuration
The configuration has to be done manually e.g. via serial cable (see the corresponding
HW Characteristics Quickguide you find on the Service CD in doc/hwinfo).
Network Design and Configuration Guide 23
4.4 Network Switch Configuration
The network switch configuration is performed via commands. The following tables shows
Cisco IOS and Cisco NX-OS configuration commands for different purposes.
Cisco Catalyst 3750 Family Configuration (IOS commands)
Basic configuration service password encryption
enable password secret
clock timezone CET 2
clock summer-time cet recurring last Sun Mar 2:00 last
Sun Oct 3:00
vtp mode transparent
udld aggressive
no ip http server
snmp-server community public RO
spanning-tree mode rapid-pvst
spanning-tree loopguard default
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
VLAN configuration for a system
with a pool pool1
vlan 10
name client-pool1
vlan 11
name storage-pool1
vlan 12
name server-pool1
vlan 13
name control
port configuration for a data port
of a rack server used as applica-
tion node
interface GigabitEthernet1/0/1
description Node rx300s6 dataNic 1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 10-12
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
Configuration Commands
24 Network Design and Configuration Guide
Cisco Catalyst 3750 Family Configuration (IOS commands)
spanning-tree portfast trunk
port configuration for a data port
of a rack server used as esx-
server
interface GigabitEthernet1/0/1
description Node rx300s6
switchport trunk encapsulation dot1q
switchport trunk native vlan 13
switchport trunk allowed vlan 10-12,20-22
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
spanning-tree portfast trunk
port configuration for a IPMI port
of a rack
interface GigabitEthernet1/0/1
description Node rx300s6 mgmtNic
switchport trunk encapsulation dot1q
switchport trunk native vlan 13
switchport trunk allowed vlan 13
switchport mode trunk
no ip address
speed auto
no mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
spanning-tree portfast trunk
channel configuration e.g. for an
uplink
interface PortChannel 1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no shutdown
interface GigabitEthernet1/0/1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no ip address
Configuration Commands
Network Design and Configuration Guide 25
Cisco Catalyst 3750 Family Configuration (IOS commands)
speed auto
mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
cdp enable
channel-group 1 mode active
no shutdown
interface GigabitEthernet2/0/1
description ISL Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
no ip address
speed auto
mdix auto
udld port
storm-control broadcast level 20
storm-control multicast level 10
cdp enable
channel-group 1 mode active
no shutdown
Save configuration copy running-config startup-config
For details see the “Catalyst 3750 Switch Software Configuration Guide” at
http://www.cisco.com.
Cisco Nexus 5000 Family Configuration (NX-OS commands)
Basic configuration no feature telnet
no telnet server enable
cfs eth distribute
feature udld
feature lacp
feature vpc
vpc domain 2
peer-keepalive destination 172.11.6.19
vrf context management
ssh key rsa 2048
udld aggressive
spanning-tree mode rapid-pvst
Configuration Commands
26 Network Design and Configuration Guide
Cisco Nexus 5000 Family Configuration (NX-OS commands)
spanning-tree loopguard default
VLAN configuration for a system
with a pool pool1
vlan 10
name client-pool1
vlan 11
name storage-pool1
vlan 12
name server-pool1
vlan 13
name control
port configuration for a data port
of a rack server used as applica-
tion node
interface Ethernet1/1
description Node rx300s6 dataNic 1
switchport trunk native vlan 11
switchport trunk allowed vlan 10-12
switchport mode trunk
spanning-tree port type edge trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
port configuration for a data port
of a rack server used as esx-
server
interface Ethernet1/1
description Node rx300s6
switchport trunk native vlan 13
switchport trunk allowed vlan 10-13,20-22
switchport mode trunk
spanning-tree port type edge trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
vpc configuration e.g. for an up-
link (same vpc number on both
nexus switches of the switch
group, per convention equal to
port-channel number)
interface port-channel2
description ISL Uplink
switchport trunk allowed vlan 1,10-13,20-22
switchport mode trunk
vpc 2
storm-control broadcast level 20
storm-control multicast level 10
no shutdown
interface Ethernet1/1
description ISL Uplink
switchport trunk allowed vlan 1,10-13,20-22
switchport mode trunk
udld aggressive
storm-control broadcast level 20
storm-control multicast level 10
Configuration Commands
Network Design and Configuration Guide 27
Cisco Nexus 5000 Family Configuration (NX-OS commands)
channel-group 2 mode active
no shutdown
Save configuration copy running-config startup-config
For details see the “Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration
Guide” at http://www.cisco.com.
API - Network
28 Network Design and Configuration Guide
5 API - Network
FlexFrame defines an API for network as described below. Providing an implementation
of this API will increase the amount of FlexFrame supported switch types by the switch
types captured from this implementation.
The implementation has to comprise the following units:
XML Description
HW Quickguide
Perl module (SwitchConfig_xxx.pm, xxx to be replaced)
Template Definitions (SwitchConfig_xxx.tpl, xxx to be replaced)
5.1 XML Description
The XML description defines the switch so far as necessary for usage by FlexFrame as
follows.
switchTypeDefinition ::=
<switch id="switchType" maxLinkAggregates="number" >
PortDefinition+
expansionModulSlotsDefinition?
groupTypeDefinitionReference+
switchMigrationDefinition?
</switch>
portDefinition ::=
<ports portFamily="portTypes" first="number" count="number">
<format type="formatType">format</format>
</ports>
expansionModulSlotsDefinition ::=
<expansionModuleSlots first="number" count="number">
expansionModulRefDefinition+
</expansionModuleSlots>
expansionModulRefDefinition ::=
<expansionModule>expansionModule</expansionModule>
referenceToSwitchgroupTypeDefinition ::=
<group id="groupType" />
switchMigrationDefinition ::=
<switchMigration>
migrateToDefinition+
API - Network
Network Design and Configuration Guide 29
</switchMigration>
migrateToDefinition ::=
<migrateTo>switchType</migrateTo>
expansionModuleDefinition ::=
<switch id="expansionModuleType">
PortDefinition+
</switch>
switchgroupTypeDefinition ::=
<switchgroup id="groupType" maxSwitches="number"
groupMgmtIPs="number" memberMgmtIPs="number"
impl="impl" domain="domain" >
managementPortDefinition?
memberConnectDefinition*
</switchgroup>
managementPortDefinition ::=
< memberMgmtPort id="interfaceName" type="portType" peer="peerType" />
memberConnectDefinition ::=
<memberConnect id="number" peer="peerType" lag="number" >
portUseDefinition+
</memberConnect>
portUseDefinition ::=
<portUse id="number" port="portID" />
FlexFrame allows the following definitions
formatType
formatType comment
1 format with one %s replacement for port number
2 format with two %s replacements, first for slot, second for port
number
needs number of slot in context
portTypes
portType comment
TX 10/100/1000 Mb/s Ethernet Ports with autosensing, RJ-45 connector
FX 1000 Mb/s SFP based Ethernet Port
API - Network
30 Network Design and Configuration Guide
TGBx 10Gb/s SFP+ based Ethernet Port
portTypes may be a colon separated list of types e.g. FX:TGBx if a port can be used with
SFP or SFP+.
peerType
peerType port should be used for connection to
AN Application Node
BB Black Box
CLAN clientLAN
CLANUPLINK clientLAN as channel
CN Control Node
ESX ESX Server
FILER NAS System
GW Gateway
KVM KVM Server
MGMT Management Interface
NONE nothing
SW Switch Group (used on Switch Blades)
SWB Switch Blade (used on Switch Group)
UPLINK Switch Group / Customer Switches
VPC_PEERLINK memberConnect for Nexus
STACKING memberConnect for ICX67xx
VCS-ISL memberConnect for VCS fabric
… memberConnect for …
A new peer type for memberConnect is introduced solely per definition in hard-
wareXML. Introduction of another new peerType must be possible solely per adding
some definitions to the SwitchConfig_xxx.tpl file.
For examples see the actual hardwareXML file in the config directory of the FlexFrame
Service DVD.
5.2 HW Quickguide
The HW Quickguide describes the manual actions to be performed when a switch will be
integrated into the FlexFrame landscape, especially how the switch will get its initial
configuration. For examples see the doc/hwinfo directory of the FlexFrame Service
DVD.
API - Network
Network Design and Configuration Guide 31
5.3 Perl Module
The Perl module is the interface to the switch handling. The Perl module includes the
template definitions file which is acting as a configuration file e.g. can be extended from
FlexFrame development when adding a new peerType. The Perl module fulfills the fol-
lowing requirements:
no persistent data
no dependencies on Perl modules other than defined in the sample
logging using methods error, warn, info and debug of logger provided by
get_logger()
Package Name
FSC_FF::SwitchConfig_xxx (xxx according hardwareXML impl value)
Class methods to be implemented
new()
return a new instance of the module
Instance methods to be implemented
register(groupCTX, groupID, switchID, attributes)
register access attributes for a switch identified by groupCTX, groupID,
switchID
return empty string if ok, error string else
Note: A switchID is unique within a group, a groupID is a cabinetID (group-
CTX = ‘cab’) or a switchGroupID (groupCTX = ‘swg’)
Note: If there is a management IP address associated to the group the method is
called only once with switchID and stype empty.
attributes = {
'name' => <name of object (may be used for file naming)>
'gtype' => <group type according XML, e.g. NEXUS5000-VPC>
'stype' => <switch type according XML, e.g. NEXUS5010>
'username' => <user to be used for login>
'password' => <password for user>
'mgmtip' => <mgmt ip for login>
}
globals(groupCTX, groupID, switchID, attributes)
request for switch globals configuration for specified object
return empty string if ok, error string else
Note: Configurations including the globals request may be applied manually us-
ing the HW Quickguide.
Note: The method is called for every object with specified mgmtip.
attributes = {
# use cn1,cn2 as ntp server, snmp trap target and syslog target
API - Network
32 Network Design and Configuration Guide
# enable snmp agent for snmp read
# create user for login as given with register
# assign mgmtip to mgmtinterface
# assign mgmtip as given with register if different
# name as given with register may be configured as prompt
'cn1' => <IP address of CN1>
'cn2' => <IP address of CN2>
'snmp' => <snmp read community>
'name' => <name of object (may be used for file naming and
prompt)>
'mgmtip' => <mgmt ip in cdr notation> e.g. 'xxx/24'
'mgmtvlan' => <vlan associated to mgmt subnet>
'domainID' => <number, e.g. domainID of NEXUS5000-VPC>
}
name(groupCTX, groupID, switchID, attributes)
request for commands to set a new name
return empty string if ok, error string else
Note: The method is called for every object with a name associated.
attributes = {
'name' => <new name (for prompt)>
}
password(groupCTX, groupID, switchID, attributes)
request for commands to set a new password for user username
returns empty string if ok, error string else
Note: The method is called for every group member regardless if necessary or
not.
attributes = {
'username' => <name of user>
'password' => <new password>
}
vlanadd(groupCTX, groupID, switchID, attributes)
request for vlan add configuration
return empty string if ok, error string else
Note: The method is called for every group member regardless if necessary or
not.
attributes = {
'vlan' => <vlanID to be added>
'name' => <vlan name to be associated, eg.'client>
}
vlanrem(groupCTX, groupID, switchID, attributes)
request for vlan remove configuration
return empty string if ok, error string else
Note: The method is called for every group member regardless if necessary or
not.
API - Network
Network Design and Configuration Guide 33
attributes = {
'vlan' => <vlanID to be removed>
}
portuse(groupCTX, groupID, switchID, portID, attributes)
request for port use configuration
return empty string if ok, error string else
attributes = {
'type' => <portfamily>, eg.'TX',
'peer' => <type of peer system>, eg.'AN',see table portPeerType
below
'desc' => <description string>,
'vlan' => <vlan ids>, eg.'t11,t12,u13'
'lag' => <link aggregate id>, eg.'5' # if port is part of a lag
'lagadd' => '1' # if lag is new to config with subsequent apply
}
portunuse(groupCTX, groupID, switchID, portID, attributes)
request for port unuse configuration
return empty string if ok, error string else
attributes = {
'vlan' => <vlan ids>, eg.'t11,t12,u13'
'lag' => <link aggregate id>, eg.'5' # if port is part of a lag
'lagrem' => '1' # if lag is obsolete to config with subsequent ap-
ply
}
portvlanadd(groupCTX, groupID, switchID, portID, attributes)
request for port vlan add configuration for a used port
return empty string if ok, error string else
Note: The method is called for every port of a lag regardless if necessary or not.
attributes = {
'vlan' => <vlan ids to be added>, eg.'t11,t12,u13'
'all' => <all vlan ids to be assigned>, eg.'t11,t12,u13'
'lag' => <link aggregate id>, eg.'5' # if port part of lag
}
portvlanrem(groupCTX, groupID, switchID, portID, attributes)
request for port vlan remove configuration for a used port
return empty string if ok, error string else
Note: The method is called for every port of a lag regardless if necessary or not.
attributes = {
'vlan' => <vlan ids to be removed>, eg.'t11,t12,u13'
'all' => <all vlan ids to be assigned>, eg.'t11,t12,u13'
'lag' => <link aggregate id>, eg.'5' # if port part of lag
}
portdown(groupCTX, groupID, switchID, portID, attributes)
request for port down configuration for a used port
API - Network
34 Network Design and Configuration Guide
return empty string if ok, error string else
attributes = {
}
portup(groupCTX, groupID, switchID, portID, attributes)
request for port up configuration for a used down port
return empty string if ok, error string else
attributes = {
}
backup(groupCTX, groupID, switchID, attributes)
request for human readable configuration for backup purposes
prefer selected configuration type else available type
tftp may be used to copy files
append timestamp to backup filename
return empty string if ok, error string else
attributes = {
'ip' => <controlLAN address of CN running tftpd –s /tftpboot>
'dir' => <directory on CN where to save configuration>
'select' => <startup | running>
}
apply(attributes)
apply previously collected requests to associated switches and forget them
save generated config commands and communication protocol to temporary di-
rectory
print messages to STDOUT/STDERR as necessary
return empty string if ok, error string else
if dryrun then suppress communication to real switches
if todo then append messages concerning manual apply according HW Quick-
guide to the referenced string instead of direct output to STDOUT
attributes = {
'dryrun' => <0|1>
'tmpdir' => <path to temporary directory>
'todo' => <reference to string>
}
Network Design and Configuration Guide 35
6 Abbreviations
cDOT Clustered Data ONTAP
DART Data Access in Real Time
DHCP Dynamic Host Configuration Protocol
FF FlexFrame
FF4S FlexFrame for SAP
FFO FlexFrame Orchestrator
IP Internet Protocol
LAN Local Area Network
MAC Media Access Control
MII Media Independent Interface
NAS Network Attached Storage
NIC Network Interface Card
ONTAP Open Network Technology for Appliance Products
PXE Preboot Execution Environment
SPOF Single Point Of Failure
TFTP Trivial File Transfer Protocol
UDP User Datagram Protocol
VLAN Virtual Local Area Network
VPC Virtual Port Channel
Network Design and Configuration Guide 37
7 Glossary
Application Node (AN)
A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This
definition includes Application Servers as well as Database Servers.
Blade
A special form factor for computer nodes.
Client LAN
Virtual network segment within FlexFrame, used for client-server traffic.
Computing Node
From the SAP ACI perspective: A host that is used for applications.
Control LAN
Virtual network segment within FlexFrame, used for system management traffic.
Control Node (CN)
A physical computer system, controlling and monitoring the entire FlexFrame land-
scape and running shared services in the rack (dhcp, tftp, ldap etc.).
Control Station
A Control Node in an SAP ACI environment.
Dynamic Host Configuration Protocol (DHCP)
DHCP is a protocol for assigning dynamic IP addresses to devices on a network.
Dynamic Host Configuration Protocol server
A DHCP server provides configuration parameters specific to the DHCP client host,
required by the host to participate on the Internet.
Ethernet
A Local Area Network which supports data transfer rates of 10 megabits per second.
Filer
Network attached storage for file systems of NetApp.
FlexFrame® (FF)
The name FlexFrame® is a generic term for both „FlexFrame
® for SAP
®“ and „Flex-
Frame® Orchestrator“.
FlexFrame® Orchestrator (FFO)
This is the advancement of the Fujitsu solution FlexFrame for SAP and means a new
approach to offer enhanced functionality and features step by step and become more
and more independent from certain hardware and software components.
FlexFrame® for SAP
® (FF4S)
FlexFrame® for SAP
® is a Fujitsu solution and means a radically new architecture for
SAP environments. It exploits the latest business-critical computing technology to de-
Glossary
38 Network Design and Configuration Guide
liver major cost savings for SAP customers. FlexFrame for SAP is a joint project in
which the main partners are SAP, Network Appliance, Intel and Fujitsu.
FlexFrame internal LAN Switch
Network switches which are integral part of the FlexFrame hardware configuration
and which are automatically configured by the FlexFrame software.
Gigabit Ethernet
A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-
bits) per second.
Host name
The name of a node (assigned to an interface) that is resolved to a unique IP ad-
dress. One node can have multiple host names (cf. node name).
In SAP environments host names are currently limited to 13 alphanumeric characters
including the hyphen (“ - “). The first character must be a letter. In the SAP environ-
ment host names are case-sensitive.
Internet Protocol Address
A unique number used by computers to refer to each other when sending information
through networks using the Internet Protocol.
Local Area Network (LAN)
A computer network that spans a relatively small area. Most LANs are confined to a
single building or group of buildings. However, one LAN can be connected to other
LANs over any distance via telephone lines and radio waves. A system of LANs con-
nected in this way is called a Wide Area Network (WAN).
Local host name
The name of the node (physical computer); it can be displayed and set using the command /bin/hostname.
Media Access Control address
An identifier for network devices, usually unique. The MAC address is stored physi-
cally on the device.
NAS system
Network Attached Storage of any vendor (in our context: NetApp Filer).
Network Attached Storage (NAS)
A data storage device that is connected via a network to one or multiple computers.
Network Interface Card (NIC)
A hardware device that allows computer communication via networks.
Node
A physical computer system controlled by an OS.
Glossary
Network Design and Configuration Guide 39
Node name
The name of a physical node as returned by the command uname -n. Each node
name within a FlexFrame environment must be unique.
Open Network Technology for Appliance Products (ONTAP)
The operating system of Network Appliance Filers.
Physical host
Name of a physical computer system (node).
Preboot Execution Environment (PXE)
An environment that allows a computer to boot from a network resource without hav-
ing a local operating system installed.
Server
A physical host (hardware), same as node.
Service
A software program providing functions to clients.
Storage LAN
A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS
systems.
Trivial File Transfer Protocol (TFTP)
A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram
Protocol (UDP) and provides no security features. It is often used by servers to boot
diskless workstations, X-terminals, and routers.
TFTP server
A simple FTP implementation.
Virtual host
The name of the virtual host on which an application runs; it is assigned to a physical
node when an application is started.
Virtual Local Area Network (VLAN)
A VLAN is a logically segmented network mapped over physical hardware according
to the IEEE 802.1q standard.
Network Design and Configuration Guide 40
8 Index
C
concept and design 5
F
FlexFrame network versions 19
enterprise version 23
medium version 19
small version 19
J
Jumbo Frames 7
L
link aggregation 5, 10, 11, 12, 16
Linux
bonding interface 25
VLAN interface 26
N
NetApp Filer configuration 26
network speed 7
network switch configuration 27
node configuration 25
R
related documents 3
V
virtual LAN 13