Inspur NF5288M5 Supercomputer White Paper · 2019-05-23 · 1 1 Product Overview Inspur NF5288M5...
Transcript of Inspur NF5288M5 Supercomputer White Paper · 2019-05-23 · 1 1 Product Overview Inspur NF5288M5...
Inspur NF5288M5 Supercomputer
White Paper
V1.5
Inspur Electronic Information Industry Co., Ltd.
Contents
1 Product Overview ............................................................................................................... 1
2 Product Features ................................................................................................................ 2
3 Product Overview ............................................................................................................... 4
3.1 Appearance............................................................................................................................... 4
3.2 Physical Structure ................................................................................................................... 5
3.3 Motherboard ............................................................................................................................. 7
3.4 GPU board ................................................................................................................................ 8
3.5 Fan module ............................................................................................................................. 10
3.6 Rear IO board ......................................................................................................................... 11
3.7 Front IO riser board ............................................................................................................... 11
3.8 Logical Structure ................................................................................................................... 12
3.9 Technical Parameters ............................................................................................................ 15
4 Module and Compatibility ................................................................................................ 18
4.1 GPU .......................................................................................................................................... 18
4.2 CPU .......................................................................................................................................... 18
4.3 Memory ................................................................................................................................... 19
4.4 Hard disk ................................................................................................................................. 21
4.4.1 Front hot swap hard disk ........................................................................................................... 21
4.4.2 Built-in hard disk ......................................................................................................................... 24
4.5 PCIe expansion card ............................................................................................................. 24
4.6 PSU .......................................................................................................................................... 26
4.7 Operating system ................................................................................................................... 27
5 BMC management system ............................................................................................... 28
5.1 Function module .................................................................................................................... 29
6 Certification and License ................................................................................................. 30
1
1 Product Overview
Inspur NF5288M5 Supercomputer (also referred to as AGX-2) is an "AI Supercomputer"
for intelligent computing, high-performance computing and accelerated video applications.
It is also a new generation 8GPU server with 2U height for Internet, intelligent security,
intelligent transportation, intelligent finance, intelligent healthcare, intelligent
manufacturing and other applications. Inspur Supercomputer NF5288M5 is the world’s
leading 2U server with 8 high-performance GPU accelerators interconnected through
300GB/s high-speed NVIDIA® NVLink ™. It is also an AI server with the world’s highest
density and best performance.
Figure 1-1 NF5288M5 External View
2
2 Product Features
Inspur Supercomputer NF5288M5 has following features:
Highest density, excellent performance:
2U standard space, 2-socket server, 8 × GPU card on NVIDIA® NVLink ™ or FHFL
dual-width PCIe GPU card.
Support 2× Intel ® Xeon ® Scalable Processor, up to 28-core processor to provide
excellent system performance, maximum 3.6 GHz base frequency, 38.5 MB L3 cache and
up to 2 × UPI at 10.4 GT/s.
Support 16 × DDR4 ECC memory of 2666 MT/s, 12 × memory channel, RDIMM and
LRDIMM, providing excellent speed, high availability, memory capacity up to 1,024GB,
and maximum memory bandwidth of 249.9375GB/s.
Support 8 × hot swap NVMe SSD and 2 × SATA M.2 SSD to maximize disk caching
rate of stored data.
Support 4 × 100G RDMA(Remote Direct Memory Access), greatly optimizing horizontal
expansion capacity.
For AI training, provide peak computing capability up to 960 Tensor TFLOPs at a single
node, over 200 times higher than traditional 2-socket server.
For HPC application, a single node delivers a peak dual-precision floating point
capability higher than 60 TFLOPs. And high-density cluster of NF5288M5 in a single
cabinet (42U) can easily reach the peak performance of more than 1 PFLOPs, (thousand
trillion of floating-point computing per second).
Support NVIDIA®NVLink2.0, allow 300GB/s bandwidth among GPUs. Hybrid Cube
Mesh reduces latency of multi-GPU data sharing, and keep cache consistency between
multiple GPUs, providing a better acceleration ratio for computing.
Flexible Configuration, Expandable as Needed
Support Tesla®SXM2 V100/P100 based on NVIDIA®NVLink ™, or V100 / P100 / P40 /
P4 GPU based on PCIe3.0 port, etc. Support 2 type interface and multiple GPU models,
allowing users to select GPU card as needed.
Connect CPU and GPU with PCIe cable, flexibly adjust CPU bandwidth and quantity,
and allocate PCIe resources more effectively as needed in different AI applications.
Flexible computing architecture allows one or two CPUs to manage up to 8 GPUs.
3
PCIe I / O and 8 × U.2 slots in the server flexibly adjust topology based on computing.
Flexible heterogeneous platforms are sufficient to support diverse AI application
scenarios.
Front panel supports 2 × PCIe3.0 X16 HHHL slots.
For SXM2 port GPU, rear panel supports 4 × PCIe3.0 X16 HHHL slots.
Support LSI3108 2G/4G cache RAID Mezz card on-board without occupying external
PCIe slot, provide customers with higher IO extensibility.
Ingenious Design and Green Innovation:
(1 + 1) redundant 3,000W 80Plus Platinum grade PSU with ultra-low power loss.
Reserve liquid cooling connectors, support gas-liquid heat radiation mode, and transfer
80% heat via GPU liquid cooling module, significantly reducing data center PUE.
Independent design and intellectual property rights of Inspur.
4
3 Product Overview
3.1 Appearance
Front panel of NF5288M5 is shown in Figure 3-1.
Figure 3-1 Front Panel
Rear panel of NF5288M5 is shown in Figure 3-2.
Figure 3-2 Rear Panel
Table 3-1 Technical Specifications for Front and Rear Panels
No. Name Quantity Description
1 PCIe I/O 2 2 × PCIe X16 HHHL front slot
2 Power module 2 80Plus Platinum PSU, 1+1
3 VGA port 1
4 USB 3.0 port 2
5 System reset button 1
5
6 System power button & LED 1
7 UID button & LED 1 Press and hold for 7 seconds to reset
BMC
8 2.5 inch hard disk holder 8
9 Rear PCIe slot 4 2 × PCIe X16 HHHL slots
(only for NVLink GPU)
10 Liquid cooling connector 1 GPU liquid-cooling connector reserved
11 C20 power connector 2 Only for C19 power cord
12 Serial port 1
13 UID button & LED 1
14 Rear VGA port 1
15 BMC management port 1
16 USB 3.0 port 2
17 10GE optical port 4 10GE optical port does not support
10/100M network. Support NC-SI
function to help you access BMC
management system via service port.
This port do not support Finisar Gen2
AOC or Rosenberger AOC cable
3.2 Physical Structure
Physical structure of NF5288M5 consists of chassis, power supply, motherboard, hard
disk, front IO riser board, fan module, GPU board and rear IO board. There are two GPU
board replacement options for NVLink GPU and PCIe GPU, to allow NF5288M5 to
support SXM2 port GPU on NVLink and standard PCIe3.0 port GPU.
NVLink GPU of NF5288M5 is shown in Figure 3-3:
6
Figure 3-3 NF5288M5 NVLink Physical Structure
PCIe GPU of NF5288M5 is shown in Figure 3-4:
Figure 3-4 NF5288M5 PCIe Physical Structure
Table 3-1 NF5288M5 Physical Structure
No. Name No. Name
1 PSU 5 System fan module
2 Hard disk 6 NVLink GPU board
3 Front IO Riser card 7 Rear IO board
7
4 Motherboard 8 PCIe GPU board
3.3 Motherboard
Motherboard of NF5288M5 is shown in Figure 3-5.
Figure 3-5 NF5288M5 Motherboard
Table 3-2 Technical Specification of NF5288M5 Motherboard
No. Name and description
1 DIMM (for CPU1)
2 Hard disk backplane power supply port and SAS, SATA signal port
3 PCIe3.0 X16 HHHL front slot
4 M.2 Riser card slot (supporting 2 × M.2 SSD hard disk)
5 ID button and ID LED & BMC reset button (Press and hold for 7 seconds to
reset BMC)
6 Power button & LED
7 System reset button
8 2*USB 3.0 port
9 VGA port
10
LSI3108 2G / 4G Cache RAID Mezz Card (supporting RAID
0/1/1E/10/5/50/6/60 with hot swap SAS/SATA hard disk, providing RAID
Cache, optional supercapacitor for data protection during power loss)
11 PCH chipset
12 Signal port (Internet, USB and VGA to rear IO board)
13 DIMM (for CPU0)
8
14 Signal port (BMC, UID and serial port to rear IO board)
15 System fan, management and signal port
16 System fan power supply port
17/26 PSU0/PSU1 power supply, management and signal ports
18/25 PSU0/PSU1 power supply port
19/24 CPU0/CPU1
20/21/22/23 2 * Slimline X8 port (each No. indicates to 2 × Slimline X8 port, one on
upper motherboard and one on lower motherboard)
3.4 GPU board
NF5288M5 support following two types of GPU boards: NVLink GPU board and PCIe
GPU board. Support two types GPU port (SXM2 port and PCIe port).
NF5288M5 NVLink GPU board is shown in Figure 3-6.
Figure 3-6 NF5288M5 NVLink GPU Board
NF5288M5 PCIe GPU board is shown in Figure 3-7.
9
Figure 3-7 NF5288M5 PCIe GPU Board
Table 3-3 Technical Specifications of NVLink GPU Board & PCIe GPU Board
No. Name and description
1/2/3/4 Slimline X8 port
5 PCIe switch (PEX9797)
6 NVLink GPU slot 0
7 NVLink GPU slot 1
8 NVLink GPU slot 2
11 NVLink GPU slot 3
12 NVLink GPU slot 6
15 NVLink GPU slot 7
16 NVLink GPU slot 4
17 NVLink GPU slot 5
9/10/13/14 PCIe X16 HHHL slot
18/29/30/39 Slimline X8 port
19/28/31/38 GPU power supply port (power supply per port to 2 GPUs)
20/21/26/27/32/33/36/37 PCIeX16 connector (to GPU Riser card)
22/23/24/25 Slimline X8 port
34/35 PCIe switch (PEX9797)
Installation sequence of PCIe GPU card is shown in Figure 3-8.
10
Figure-3-8 PCIe GPU Card Sequence
3.5 Fan module
NF5288M5 uses 5 pairs of dual rotor pairs of rotating redundant system fan, N+N
redundant mode. When 1 rottors are broken in each group of fans, the fan group exhaust
volume can be kept constant
NF5288M5 fan module is shown in Figure 3-9.
Figure-3-9 System Fan Module
Table 3-4 Technical Specification of System Fan Module
No. Name and description
1 5 × dual-rotor redundant system fan
GPU0 GPU2 GPU1 GPU3 GPU4 GPU5 GPU6 GPU7
11
2 System fan power supply port
3 System fan, management and signal port
3.6 Rear IO board
NF5288M5 rear IO board is shown in Figure 3-10.
Figure 3-10 Rear IO Board
Table 3-5 Technical Specification of Rear IO Riser Board
No. Name and description
1 Serial port
2 ID button & LED
3 VGA port
4 Management port
5 2*USB port
6 4×10G SFP+ Ethernet (supporting NC-SI from left to right eth0 - eth3,
helping you access BMC management system via service network)
7 Signal port (Internet, USB and VGA to motherboard)
8 Signal port (BMC, UID and serial port to motherboard)
3.7 Front IO riser board
Front IO Riser board of NF5288M5 is shown in Figure 3-11.
12
Figure 3-11 Front IO Riser Board
Table 3--7 Technical Specification of Front IO Riser Board
No. Name and description
1 PCIeX16 slot (2 × Slimline X8 is converted into PCIE signal)
2 2 Slimline X8 interface (to motherboard or PCIe GPU board)
3 PCIe X16 slot (Motherboard PCIe switch)
3.8 Logical Structure
Motherboard logical block diagram of NF5288M5 is shown in Figure 3-12.
13
Figure 3-12 NF5288M5 Motherboard Logical Block Diagram
NVLink GPU board logical block diagram of NF5288M5 is shown in Figure 3-13:
14
Figure 3-13 NF5288M5 NVLink GPU Board Logical Block Diagram
PCIe GPU board logical block diagram of NF5288M5 is shown in Figure 3-14:
15
Figure 3-14 NF5288M5 PCIe GPU Board Logical Block Diagram
3.9 Technical Parameters
Complete technical parameters of NF5288M5 are listed in Table 3-8.
Table 3-8 Technical Parameters
Module Name Specification description
Form 2U rack server
Processor 1/2 Intel® Xeon® 3100, 4100, 5100, 6100, 8100 series
Scalable Processor:
Support up to 28 cores (2.5GHz frequency)
Maximum frequency 3.6GHz (4 cores)
Support L3 cache up to 38.5MB (28 cores)
2 × UPIs with maximum rate 10.4GT/s per link
Maximum thermal design power 165W.
Chipset Intel® C624 chipset
Memory Every CPU support 8 x DIMM, total 16 x DIMM
Every CPU support 6 x memory channel, total 12 x memory
channel
IO port 2 front set USB 3.0 port, 1 VGA 1 ID button & ID LED & BMC
reset button, 1 system power button & LED, 1 system reset
button
16
2 rear set USB 3.0 port, 1 VGA, 1 serial port, 1 ID button &
LED, 1 BMC management port
RAID card Support 1pcs 8-channel high-performance SAS RAID Mezz
card without occupying standard PCIe slot, improving system
expansibility.
Optional super-capacitor protection module, support Raid
0/1/10/5/50/6/60.
Storage Front 8 × 2.5 inch hot swap SAS / SATA / U.2 port NVMe SSD
hard disks; 2× M.2(SATA) SSDs built-in
On-board Network Integrated Intel X722 provide 4*10GE optical port, support
NCSI, Wake-on-LAN, network redundancy, load balancing
and other advanced features
PSU 2pcs 3,000W 80Plus Platinum PSU (Redundant under
restrictive condition)
Support 100V-240V AC and 240V DC input
C20 power port (with C19 power cord)
Heat radiation 5 × redundant hot swap system fan,
Reserved NVLink GPU liquid cooling connector
GPU For NVLink GPU board:
8 × SXM2 port, support NVIDIA Tesla V100 / P100 (SXM2) on
NVLink, 4 × PCIeX16 HHHL rear slot
For PCIe GPU board:
8 built-in GPU PCIe3.0 X16 FHFL and dual-width slot, support
NVIDIA Tesla V100 (PCIe) /P100 (PCIe) /P40/P4, etc.
PCIe I/O For PCIe GPU:
2 × PCIe 3.0 X16 HHHL front slot
For NVLink GPU:
2 × PCIe 3.0 X16 HHHL front slot and 4 × PCIe 3.0 X16 HHHL
rear slot
Monitor and
Management
Provide specific Ethernet port for management. Supports
remote monitoring, SMTP KVM, SNMP management, Virtual
Media and redundant management system
Support OS Windows server 2012 R2/2016
Redhat Linux Enterprise Server 7.2/7.3/7.4
Suse Linux Enterprise Server 12 and other main operating
systems
Chassis Size 448mm width × 87.5mm height × 899.5mm depth (17.64in
width × 3.44in height × 35.41in depth)
Net Weight ≤48kg
Operating Environment
Temperature 10℃ to 35℃
17
Transportation &Storage
temperature -40℃ to 60℃
Operating humidity 20%-80% RH
Storage humidity 20%-93% RH
Storage &Operating
altitude ≤3000M
18
4 Module and Compatibility
4.1 GPU
Support 0/4/8 ×standard PCIe GPU card. Or
Support 0/4/6/8 × SXM2 NVLink GPU card.
Support up to 8 × NVLink NVIDIA® TESLA® V100 / P100 GPU card
Support up to 8 PCIe GPU cards such as NVIDIA® TESLA® V100 / P100 / P40 / P4,
etc.
supports both NVIDIA® NVLink ™ 1.0 and NVIDIA® NVLink ™ 2.0
Maximum P2P bandwidth 300GB/s
Support up to 300W GPU TDP
For SXM2 interface NVLink-connection GPU card, NVLink GPU board should be
configured, supported GPUs are shown in Table 4-1.
Table 4-1 NVLink GPU Compatibility List
GPU No. GPU model
X01700D000000000 GPU_NV_16G_Tesla-P100_4096b_S_CAC
X01700K000000000 GPU_NV_16GB_Tesla-V100_4096b_S_SXM2
For standard PCIe interface GPU card, PCIE GPU board should be configured, supported
GPUs are shown in Table 4-2.
Table 4-2 PCIe GPU Compatibility List
GPU No. GPU model
X01700B000000000 GPU_NV_12G_TESLA-P100_4096b_P_CAC
X01700C000000000 GPU_NV_16G_TESLA-P100_4096b_P_CAC
X01700F000000000 GPU_NV_24G_TESLA-P40_384b_P_CAC
X01700G000000000 GPU_NV_8G_P4_256b_P_CAC
X01700L000000000 GPU_NV_16GB_Tesla-V100_4096b_P_CAC
4.2 CPU
Support 1/2 Intel® Xeon® 3100, 4100, 5100, 6100, 8100 series Scalable Processor:
Support up to 28 cores (2.5GHz frequency)
19
Maximum frequency 3.6GHz (4 cores)
2 × UPIs with maximum rate 10.4GT/s per link
Support L3 cache up to 38.5MB (28 cores)
Maximum thermal design power 165W
For 2 × CPUs, CPUs are installed on CPU0 and CPU1 slots of the motherboard. For only
1× CPU, CPU is installed on CPU0 slot.
CPU supported by NF5288M5 is shown in Figure 4-3.
Table 4-3 CPU Compatibility List
CPU No. CPU model
V00103P000000000 CPU_I_4114-Xeon2.2_10_9.6_13.75_85
V00103V000000000 CPU_I_4116-Xeon2.1_12_9.6_16.5_85
V00102J000000000 CPU_I_5117-Xeon2.0_14C_19.25M_105W_BD
V00103N000000000 CPU_I_5118-Xeon2.3_12_10.4_16.5_105
V001037000000000 CPU_I_6126-Xeon2.6_12_10.4_19.25_125
V001039000000000 CPU_I_6130-Xeon2.1_16_10.4_22_125
V00103W000000000 CPU_I_6132-Xeon2.6_14_10.4_19.25_140
V00102U000000000 CPU_I_6134-Xeon3.0_8_10.4_24.75_130
V00102T000000000 CPU_I_6138-Xeon2.0_20_10.4_27.5_125
V001045000000000 CPU_I_6138-Xeon2.0_20C_27.5M_125W
V001030000000000 CPU_I_6140-Xeon2.3_18_10.4_24.75_140
V00102R000000000 CPU_I_6142-Xeon2.6_16_10.4_22_150
V001032000000000 CPU_I_6148-Xeon2.4_20_10.4_27.5_150
V001031000000000 CPU_I_6152-Xeon2.1_22_10.4_0.25_140
V00102V000000000 CPU_I_8160-Xeon2.1_24_10.4_33_150
V00102K000000000 CPU_I_8163-Xeon2.5_24C_33M_165W_ALI
V00102Z000000000 CPU_I_8164-Xeon2.0_26_10.4_35.75_150
V001038000000000 CPU_I_8170-Xeon2.1_26_10.4_35.75_165
V001034000000000 CPU_I_8176-Xeon2.1_28_10.4_38.5_165
4.3 Memory
NF5288M5 has 16 × DDR4 DIMM slots. Support DDR4 2133/2400/2666MT/s
RDIMM/LRDIMM. Each processor provides 6 memory channels, in which CPU0_C0,
CPU0_C3, CPU1_C0, CPU1_C3 support 2 × DIMMs.
NF5288M5 supports a single DIMM of memory capacity 16GB, 32GB and 64GB,
maximum memory support is up to 1024GB.
20
Memory installation rules are shown in table 4-4 and table 4-5.
Table 4-4 NF5288M5 Dual-CPU Memory Configuration
DRAM supported by dual
CPUs
Memory quantity and memory installation location
2 4 6 8 10 12 14 16
DRAM
supported
by CPU0
CPU0_C0D0 ● ● ● ● ● ● ● ●
CPU0_C0D1 ● ●
CPU0_C1D0 ● ● ● ● ● ● ●
CPU0_C2D0 ● ● ● ● ● ●
CPU0_C3D0 ● ● ● ● ●
CPU0_C3D1 ●
CPU0_C4D0 ● ● ● ●
CPU0_C5D0 ● ● ●
DRAM
supported
by CPU1
CPU1_C0D0 ● ● ● ● ● ● ● ●
CPU1_C0D1 ● ●
CPU1_C1D0 ● ● ● ● ● ● ●
CPU1_C2D0 ● ● ● ● ● ●
CPU1_C3D0 ● ● ● ● ●
CPU1_C3D1 ●
CPU1_C4D0 ● ● ● ●
CPU1_C5D0 ● ● ●
Table 4-5 NF5288M5 Single-CPU Memory Configuration
DRAM supported by a
single CPU
Memory quantity and memory installation location
1 2 3 4 5 6 7 8
DRAM
supported
by CPU0
CPU0_C0D0 ● ● ● ● ● ● ● ●
CPU0_C0D1 ● ●
CPU0_C1D0 ● ● ● ● ● ● ●
CPU0_C2D0 ● ● ● ● ● ●
CPU0_C3D0 ● ● ● ● ●
CPU0_C3D1 ●
CPU0_C4D0 ● ● ● ●
CPU0_C5D0 ● ● ●
21
● Memory installation rules:
One server supports only one memory type. Memory installation rules are described as
follows:
A. With priority to white slot, CPU1 and CPU0 memory needs to be symmetrically
installed.
B. In case of single CPU, memory is installed in the following order: CPU0_C0D0,
CPU0_C1D0, CPU0_C2D0, CPU0_C3D0, CPU0_C4D0, CPU0_C5D0; CPU0_C0D1,
CPU0_C3D1
C. In case of double CPUs, memory at CPU0 is installed in the following order:
CPU0_C0D0, CPU0_C1D0, CPU0_C2D0...; memory at CPU1 needs to be
symmetrical with memory at CPU0: CPU1_C0D0, CPU1_C1D0 and CPU1_C2D0...
DIMM position of NF5288M5 is shown in Figure 4-1.
Figure 4-1 NF5288M5 DIMM Slot Position Diagram
4.4 Hard disk
4.4.1 Front hot swap hard disk
NF5288M5 supports front 8 × 2.5 inch SAS / SATA / NVMe hard disk, and mix use with
multi-capacity and multi-type hard disk.
For SAS / SATA hard disk, hard disk sequence at delivery is shown in Figure 4-2.
22
Figure 4-2 SAS / SATA Hard Disk Configuration
There are two types of hard disk backplane, 1 backplane support 4 hard disk and support
up to 2 backplanes. as shown in Table 4-6.
Table 4-6 Hard Disk Backplane List
No. Backplane name
YZBB-00906-101 Backplane_Inspur_Vancouver_HDDBP_SATA_NVMe
YZBB-00835-101 Backplane_Inspur_Vancouver_HDDBP_SAS_NVMe
YZBB-00906-101 supports 2.5 inch SATA hard disk. SATA signal port is connected
from PCH chip on motherboard. With NVMe cable, supports NVMe hard disk, SATA and
NVMe hard disks in any capacity and number, and software RAID0/1/5/10. This
configuration does not need RAID Mezz card.
YZBB-00835-101 is used with Inspur 3108 RAID Mezz card, and supports 2.5 inch
SATA / SAS hard disk. SAS signal port is connected from RAID Mezz card, supports Raid
0/1/10/5/50/6/60 and RAID card direct pass-through mode; with NVMe cable, support
NVMe hard disk, and support mix use of SATA/SAS and NVMe hard disk in any capacity
and quantity.
SAS/SATA hard disk compatibility is shown in Table 4-7.
Table 4-7 SAS/SATA Hard Disk Compatibility List
Type Name
SAS_HDD_2.5 300G_15k_12G rate_2.5 inch SAS hard disk
300G_10k_12G rate_2.5 inch SAS hard disk
450G_10k_12G rate_2.5 inch SAS hard disk
450G_15k_12G rate_2.5 inch SAS hard disk
600G_15k_12G rate_2.5 inch SAS hard disk
600G_10k_12G rate_2.5 inch SAS hard disk
900G_10k_12G rate_2.5 inch SAS hard disk
900G_15k_12G rate_2.5 inch SAS hard disk
1T_7.2k_12G rate_2.5 inch SAS hard disk
1.2T_10k_12G rate_2.5 inch SAS hard disk
23
1.8T_10k_12G rate_2.5 inch SAS hard disk
2T_7.2k_12G rate_2.5 inch SAS hard disk
SATA_HDD_2.5 1TB_7.2k_6G rate_2.5 inch SATA hard disk
2TB_7.2k_6G rate_2.5 inch SATA hard disk
SATA_SSD_2.5 150GB_SATA SSD hard disk
240GB_SATA SSD hard disk
400GB_SATA SSD hard disk
480GB_SATA SSD hard disk
(support Micron 5100 series)
800GB_SATA SSD hard disk
960GB_SATA SSD hard disk
1.2TB_SATA SSD hard disk
1.6TB_SATA SSD hard disk
1.92TB_SATA SSD hard disk
(support Intel S4600 series)
3.8TB_SATA SSD hard disk
For full NVMe hard disk configuration, hard disk sequence at delivery is shown in Figure
4-2.
Figure 4-3 NVMe Hard Disk Configuration
NVMe hard disk is used with built-in NVMe hard disk cable.
While doing RAID0/1/10 for NVME hard disk,RAID Key code is V08902F000000000
Raidkey_I_Stand_RAID0-1-10_Black
While doing RAID0/1/5/10 for NVME hard disk,RAID Key code is V08902E000000000
Raidkey_I_Prem_RAID0-1-5-10_Red
For NVLink GPU card and add-in card in upper front PCIe slot, only support 4 NVMe
hard disks.
For 1pcs CPU and NVLink GPU card, front NVMe hard disk is not supported.
For a single processor a PCIe GPU and add-in card in upper front PCIe slot, only
support 4 NVMe hard disks.
24
NVMe hard disk compatibility is shown in Figure 4-8.
Table 4-8 NVMe Hard Disk Compatibility List
Type Name
NVMe SSD
960GB_U.2 port NVMe hard disk
1TB_U.2 port NVMe hard disk
2TB_U.2 port NVMe hard disk
4TB_U.2 port NVMe hard disk
1.9TB_U.2 port NVMe hard disk
3.8TB_U.2 port NVMe hard disk
4.4.2 Built-in hard disk
NF5288M5 supports optional built-in 1 or 2 × M.2 SATA SSD with M.2 Riser card. building
soft RAID under BIOS, and RAID 0/1 mode, but does not support mixing two different M.2
SSDs.
M.2 SSD compatibility is shown in Figure 4-9.
Table 4-9 M.2 SSD Compatibility List
Type Name
M.2 port SATA bus SSD
80GB_M.2 port SATA bus SSD
120GB_M.2 port SATA bus SSD
150GB_M.2 port SATA bus SSD
240GB_M.2 port SATA bus SSD
340GB_M.2 port SATA bus SSD
480GB_M.2 port SATA bus SSD
760GB_M.2 port SATA bus SSD
960GB_M.2 port SATA bus SSD
4.5 PCIe expansion card
In addition to GPU card, NF5288M5 also supports PCIe extension cards with different
types and rates.
PCIe card collocation is as follows:
RAID Mezz card is installed on motherboard without occupying external PCIe slot.
For NVLink GPU, support up to 6 × external PCIe slot.
25
For PCIe GPU, support up to 2 × external PCIe slot.
For NVLink GPU configuration, support 4 × PCIe X16 HHHL rear, and 100G InfiniBand
card RDMA.
For 1 processor and NVLink GPU, support 4 x PCIE, do not support front PCIE and
NVME hard disk.
For 2 processors, NVLink GPU and 8 × NVMe hard disk, do not support front upper
PCIe slot.
For 1 processor and PCIE GPU, support front upside PCIE and 4 x NVME without front
downside PCIE support, or support 8 x NVME without front PCIE support.
Support mixing of Ethernet card, IB card, and OPA card support in any type and
quantity.
Current PCIe expansion card compatibility is shown in Table 4-10.
Table 4-10 PCIe Expansion Card Compatibility List
Type Name
RAID card
Inspur self-developed 2GB cache 3108 RAID Mezz card (support RAID or
direct pass-through)
Inspur self-developed 4GB cache 3108 RAID Mezz card (support RAID or
direct pass-through)
Ethernet
card
Mellanox_100G one-port network card (with optional 100G multi-mode optical
module)
Mellanox_100G two-port network card (with optional 100G multi-mode optical
module)
Mellanox_40G one-port network card(with optional 40G multi-mode optical
module)
Intel_XL710_40G one-port network card
Mellanox _25G one-port network card
(with optional 25G multi-mode optical module or special fixed-length cable with
optical module)
Mellanox _25G dual-port network card (with optional 25G multi-mode optical
module)
(with optional 25G multi-mode optical module or special fixed-length cable with
optical module)
Inspur self-developed 82599_10G dual - optical port network card (with
optional 10G SFP+ single-mode/multi-mode optical module)
Mellanox_10G dual - optical port network card (with optional 10G SFP+
single-mode/multi-mode optical module)
26
Intel_82599_10G dual - optical port network card
(including optical module)
I350 1G dual port RJ45
Intel X540 10G dual port RJ45
IB card
Mellanox EDR IB (100Gb/s) single-port card (with optional special fixed-length
cable including optical module)
Mellanox EDR IB (100Gb/s) dual-port card (with optional special fixed-length
cable including optical module)
Mellanox IB 56Gb/s single-port card(with optional special fixed-length cable
including optical module)
OPA card Intel OPA 100Gb/s single-port card (with optional special fixed-length cable)
4.6 PSU
NF5288M5 configure 2 power modules as default. While 8pcs 250/300W GPU configured
and running NVqual under heavy power test, the power can not meet 1+1 redundancy.
But under most training model of deep learning the power have 1+1 redundancy. Optional
power module available are shown in Table 4-11.
Table 4- 11 Power Supply Specification
Power mode Rated power Rated input voltage
3000W 80Plus Platinum PSU 3000W 100V AC - 240V AC
Ready for 240 v DC
Note: Standard G16-C19 interface or C19-C20 interface 16A AC power cord is needed.
And every power cord must need 1pcs power cord filter.
Power cord filter for NF5288M5 is shown in table 4-12.
Table 4-12 power cord filter
Model of power cord filter note
power filter_GW_16A_GW-CDM3000AA-EMI Match standard G16-C19 or C19-C20
interface 16A AC power cord,And every
power cord must need 1pcs power cord
filter.
Power cord filter is a low-pass filter that allows DC or 50Hz working current to
pass through, and does not allow high frequency EMI signals to pass through.
Power cord filter is bidirectional, not only prevent interference from the power
grid go into the equipment to produce adverse effects on the equipment,
making equipment meet requirements of conduction sensitivity ; but also
prevent EMI from the equipment go into the power grid through the power cord,
making equipment meet requirements of conduction emission.
27
4.7 Operating system
Operating system options supported by NF5288M5 are shown in Figure 4-12.
Table 4-12 Operating System Compatibility
part/mode
OS
RAID
card_S
AS3108
_2GB
RAID
card_S
AS3108
_4GB
sSATA
AHCI
mode
sSATA
RAID
mode
SATA
AHCI
mode
SATA
RAID
mode
NVMe
single
disk
mode
NVMe
RAID
mode
Microsoft Windows
Server 2012 R2 UEFI √ √ √ √ √ √ √ √
Microsoft Windows
Server 2012 R2
Legacy
√ √ √ √ √ √ N/A N/A
Microsoft Windows
Server 2016 UEFI √ √ √ √ √ √ √ √
Microsoft Windows
Server 2016 Legacy √ √ √ √ √ √ N/A N/A
Red Hat Enterprise
Linux 7 U3 64Bit UEFI √ √ √ √ √ √ √ √
Red Hat Enterprise
Linux 7 U3 64Bit
Legacy
√ √ √ √ √ √ N/A N/A
Red Hat Enterprise
Linux 7 U4 64Bit UEFI √ √ √ √ √ √ √ √
Red Hat Enterprise
Linux 7 U4 64Bit
Legacy
√ √ √ √ √ √ N/A N/A
CentOS 7.3 UEFI √ √ √ √ √ √ √ √
CentOS 7.3 Legacy √ √ √ √ √ √ N/A N/A
Ubuntu 14.04 UEFI √ √ √ √ √ √ √ √
Ubuntu 14.04 Legacy √ √ √ √ √ √ N/A N/A
Ubuntu 16.04 UEFI √ √ √ √ √ √ √ √
Ubuntu 16.04 Legacy √ √ √ √ √ √ N/A N/A
28
5 BMC management system
NF5288M5 integrates a new generation Inspur server BMC management system that is a
self-developed by Inspur for remote management of server. It conforms to server industry
management standard IPMI 2.0, and possesses highly reliable hardware monitoring and
management functions.
Main functions and features of Inspur server BMC management system are listed as
follows:
● Remote control
Server control is realized via functions such as KVM (Keyboard Video and Mouse), SOL
(Serial Over Lan) and virtual media.
Note: SOL function has to be realized via third-party tools such as IPMI Tool.
● Alarm management
Report alarm information in real time, and deal with it accordingly.
● Condition monitor
Monitor various operating conditions of each monitoring unit in real-time.
● Equipment information management
Provide equipment version information, model and asset information function.
● Heat radiation control
Adjust fan speed dynamically according to ambient temperature and workload.
● Support IPMI Tool management
Support commands from IPMI Tool. Download IPMI Tool at will.
Note: IPMI Tool download website: http://ipmitool.sourceforge.net/manpage.html
● Support WEB interface
Provide friendly visual interface management, allowing quick setting and query by clicking.
● Support centralized account management
Support centralized account storage in Active Directory server, complete authentication
process in server, and allow domain account to log in management system.
29
5.1 Function module
Inspur Server BMC management system consists of IPMI module, command line module,
WEB module, KVM Over IP, virtual media, etc.
● Command line module allows calling IPMI module. Users can operate IPMI module via
command line.
● WEB module completes routine management of server in the form of a visual interface
by calling IPMI command, integrates KVM and virtual media.
IPMI module:
IPMI module completes the server system management function according to IPMI2.0
standard.
IPMI module support following functions:
● Real-time system monitor
In case of failure detected, report and indicate alarms.
● Remote control of system
Remotely power on/off and reset system, etc. via command line and Web.
Command line module:
Command line module consists of query and setup commands of network, sensor, fan,
user management, system, server, etc.
Remote control module:
●KVM Over IP: Users monitor and control remote devices by using local video, keyboard
and mouse on client, so as to operate remote devices in real time.
● Virtual media: Provide remote access to local media (CD-ROM, floppy disk drive, or
compact / floppy disk mirror image file) by means of virtual CD drive and floppy drive on
server.
Remote control function needs corresponding browser version and Java operating
environment.
Note:
If Java operating environment fails to meet requirements, download from
http://www.oracle.com/technetwork/java/javase/downloads/index.html
30
6 Certification and License