Engineer Innovate Integrateengineer innovate integrate

Quentin Le Burel

CIUK 2016

Engineer Innovate Integrateengineer innovate integrate

About ClusterVision

• End-to-end Cloud, Big Data & HPC solution provider

• 14 years Europe's Leading Specialist for High-Performance Computing

• Expertise in HPC and Cloud technologies

• Open source advocate with Enterprise level support

• Proud member of ETP4HPC (European Technology Platform)

• Active in Europe, Middle-East & Africa

• Offices in United Kingdom, Germany, France and The Netherlands

• High Quality: ISO9001:2008 & ISO14001 certified

Engineer Innovate Integrateengineer innovate integrate

ClusterVision References EMEA

Government EducationIndustry

Engineer Innovate Integrateengineer innovate integrate

ClusterVision Leading Technologies

Intel SkyLake CPU

� “Tock” step: New architecture

� AVX 3.1: 512 bits vectors (Haswell: 256)

� 1.5x Memory BW, 2x Linpack

� Possible Mesh interconnect support

� Omnipath integrated: no PCI-E overheads

� ≥24 core count per CPU

BeeGFS

� Superfast parallel & reliable filesystem

� More POSIX compliant than NFS

� Scales linearly with number of storage units

� Failover capability using multiple servers

� Over 80 large installations, total 40PB of data

LSI Syncro 9361/8 + SMC SuperChassis

� HA Dual Controller RAID protection

� Cache mirroring by default

� 9 Gbyte/s read, 5 Gbyte/s, 12G SAS

� RAID 0, 1, 5, 6, 10, 50, 60

� CacheVault: Cache Battery protected

� CacheCade: SSD Caching up to 512GB

� 90x 3.5’’ Top Loading SAS3 12Gb/s

� Tool-less HDD trays

� Perfect for Lustre/BeeGFS/GPFS

Intel Omnipath Interconnect

� Replacement for Intel Truescale Infiniband

� 100GB/s bandwidth

� 48 port switches -> higher density

� 0.7us latency (est.)

� PCIe and later CPU integrated

� Compatible and consistent with Truescale APIs

Engineer Innovate Integrateengineer innovate integrate

ClusterVision Development - TrinityX

Integration with existing environments

� Complete HPC user environment (modules environment,

scientific libraries, compilers, profilers, debuggers)

� MPI libraries - OpenMPI, Intel MPI, and optional

mpich/mvapich/mvapich2

� SLURM preconfigured to make full use of a cluster.

Optional support for PBS, Torque, Moab, LSF & SGE.

� Full HPC performance using the optional Docker-based

application containerization

� Parallel filesystem support: Lustre, IBM Spectrum Scale

(GPFS), and BeeGFS.

TrinityX functionality

� Lightning-fast Torrent provisioning tool can get 400

nodes up and running in under 5 minutes

� Full hardware integration (Infiniband, Omnipath, PXE,

Intel Broadwell, IPMI, switches etc)

� A comprehensive monitoring and metering system to

keep track of critical events and resource usage

� HA for controllers, storage, and login nodes

� Node -> switch integration, automatic discovery

� An integrated authentication system that can be

plugged to existing backends with minor tweaks

(LDAP)

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Parallel Filesystem specs

Fully HA, redundant storage

▪ All disks RAID protected (RAID6 for object, RAID1 for meta)

▪ Dual expander SuperMicro JBODs

▪ One pair of Avago Syncro cards per JBOD

▪ 2 pairs of Huawei servers

▪ Servers connected to different IB Edge switches

▪ IBM Spectrum Scale (GPFS) filesystem

Top level performance

▪ 1PB of usable storage

▪ 15GB/s sustainable read/write

▪ 10k IOPS

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - NVMe storage specs

In each node:

▪ 10x Intel P3600 NVMe PCIe 3.0 SSDs drives in a RAID10 (striped mirrors)

Each disk: Seq R/W : Up to 2600/1700MB/s

▪ System is network limited !

EDR network: 10GB/s

Total:

▪ 12TB of usable storage

▪ 20GB/s sustainable read/write

▪ 4.5M IOPS read

▪ 500k IOPS write

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - GPFS distributed storage

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - GPFS distributed storage

Engineer Innovate Integrateengineer innovate integrate

Questions ?

Engineer Innovate Integrateengineer innovate integrate

ClusterVision Engineering

Infrastructure integration and testing

� Racking & Stacking at location

� Electrical & Cooling infrastructure integration

� Local Network and LDAP user access integration

� Stress testing: Long HPLs, Streams, IMB-MPI, hardware

reboots, etc.

� SMART/IPMI/MCE Linux testing

� Team of dedicated HPC specialists for Support

Big Data, Cloud and HPC techniques

OpenStack, Beegfs, Ceph, IBM GPFS, NFS, Lustre, XFS, BTRFS, EXT4, ZFS,

NFSoIB, RDMA, OpenMPI, Mvapich, Intel MPI, GCC, Intel Compilers,

Mellanox, Truescale, Omnipath, Openstack, Arista, Containers, KVM,

VMWARE, Hadoop, Lapack, Scalapack, HDF, Blas, Blacs, Slurm, Slurm,

SGE, PBS, Torque, xcat, IPMI, 10/40/100 Gbit/s, LDAP, IBM Platform,

Modules Environment, Redhat, CentOS, Scientific Linux, Samba, NVDIA

Tesla, AMD Firepro, Intel Knights Landing, ScaleMP, Intel Xeons/Atoms,

ARM X-Gene, AMD APU, APC cooling tunnels & PDUs, Backdoor cooling,

Oil cooling, Nagios, Ganglia, Bright Cluster Manager, LSI Syncro,

Mellanox, ScaleMP, NICE, Asus, Supermicro, Samsung, Hynix, QDR, FDR,

EDR, FDR10, Western Digital, Centec, Quanta, LSI, Kingston, PCIe SSD,

DNS, IPoIB..

Designed for reliability: HA services

� Load Balancing: Round Robin DNS Login nodes

� Redundant PSUs, Networking, Infiniband

� Redundant controller nodes: Q-system & Monitoring

� Highly available DATA: NFS, GPFS & Lustre

COMPACT

RESET

1 2

UPERMICRS

COM PACT

RESET

1 2

UPERMICRS

Engineer Innovate Integrateengineer innovate integrate

ClusterVision Innovation

OpenStack, BigData & HPC: TrinityX

▪ Convergence of big data, private cloud and HPC

▪ Open industry standard, OpenSource

▪ No HPC Performance loss: Linux Containers

HPC mineral oil based cooling solution

▪ Saving <20% power: no air-cooling or fans

▪ Less current leakage, higher HPC performance

▪ Skinless servers (ClusterVision designed)

▪ Re-use of racks and same oil for > 15 years

▪ Improved Hardware Life: lower peak temperature

Remote System Administration

▪ Outsourced infrastructure management

▪ Power of scaling: lower your cost

▪ Especially suitable for OpenStack

▪ You can focus on users instead of hardware

Enterprise level services organisation

▪ Professional Support on OpenSource products

▪ Flexible On Demand Service Credits

▪ Standard workpackages

▪ Consulting, training, technology integration,

application fine-tuning, code porting

Engineer Innovate Integrateengineer innovate integrate

ClusterVision Development - TrinityX

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Infiniband EDR Network

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - NVMe storage for OpenPower nodes

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Parallel Filesystem

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Parallel Filesystem (Zoom)

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Multi-tier policy

NVMe storage:

▪ Small storage (10TB)

▪ Very fast

▪ Expensive !

▪ No HA

▪ Dedicated for OpenPower nodes, less users

Syncro/SuperMicro storage:▪ Large storage (1PB)

▪ Fast

▪ Cheaper (per TB)

▪ HA

OpenPower nodes:▪ Massive amount of data

▪ Reading data

▪ Writing data

▪ Should not impact main storage system

Engineer Innovate Integrateengineer innovate integrate

ClusterVision - Case Study - Multi-tier policy