No. 200January 2019 Bits & Bytes

Max Planck Computing & Data Facility (MPCDF)∗

Gießenbachstraße 2, D-85748 Garching bei München

Bits & Bytes turns 50!

Hermann Lederer, Andreas Schott

The first edition of Bits & Bytes appeared in March 1969.The occasion at that time was the installation of a pow-erful IBM 360/91 system delivered on January 2, 1969at the Institute for Plasma Physics (IPP) in the newlybuilt machine hall. The IBM 360/91 showed an incred-ible performance increase of 100 compared to the IBM7090 installed in 1962 – which was also the most power-ful machine worldwide at that time.

Figure 1: IBM 360/91 in the new machine hall 1969

Figure 2: Cabling of IBM 360/91 (1969)

Moore’s law – formulated 1965 – was verified by the in-credible performance increase of supercomputers in thelast fifty years at the IPP and the Max Planck Society: afactor of 1 billion from 10 MFlop/s of the IBM 360/91 in1969 to 10 PFlop/s of the current Max Planck supercom-puter Cobra in 2019, with more than 100,000 computecores. Bits & Bytes reflects well the progress in compute,storage, network and programming technologies of thelast half century. It documents both dramatic changesas well as things that have remained constant. The June1969 edition of Bits & Bytes reports about heavy fightsamong opponents and supporters of the Fortran program-ming language, stating: "The style reminds of medievaltheologic debates". Wide area network (WAN) connec-tions existed to Munich Max Planck Institutes, and toKarlsruhe (KFK) a 40 kbit/s connection. Compared totoday’s aggregated WAN capacity of around 200 Gbit/sat our centre, we observe a more than million-fold in-crease. A hundred editions of Bits & Bytes later, fourdecades ago, two further major development steps oc-curred: Firstly, the installation of a Cray-1 system at theIPP as the worldwide first vector computer for generalbasic research which enabled the development of the ba-sic features of the Stellarator fusion device Wendelstein7-X. Secondly, with the installation of the first automaticrobotic tape system, a CDC 38500, with a capacity of 16GB based on 2000 cartridges with 8 MB each, the era ofmass storage started at the Rechenzentrum Garching –as our centre was officially christened at that time. Thename lasted three and a half decades until the renaming toMax Planck Computing and Data Facility (MPCDF) fouryears ago, emphasizing now also the key importance ofdata.The MPCDF has meanwhile advanced into the top10 academic data centres worldwide listed in the websiteof the High-Performance Storage System HPSS, and isthe first placed German institute, with more than 100 PBof stored data – on around ten times the number of tapeswith a capacity increase per tape of up to 1.5 million (12TB for LTO-8). We hope you enjoy reading this jubileeedition 200 of Bits & Bytes.

∗Tel.: +49(89) 3299-01, e-mail: benutzerberatung@mpcdf.mpg.de, URL: https://www.mpcdf.mpg.de/Editor: Renate Dohmen

Bits & Bytes, No. 200, January 2019 2

High-performance Computing

Ingeborg Weidl, Hermann Lederer, Markus Rampp

Supercomputer Cobra extended with VoltaGPUs

The Max Planck supercomputer Cobra has been extendedwith GPU-accelerated nodes. 64 new nodes (with In-tel Skylake processors) equipped with two NVIDIA V100Volta GPUs each were added. The V100 GPUs provide32 GB of high-bandwidth memory (HBM) each.

Decommissioning of the Hydra Cluster

After more than 5 years of operation, the HPC clusterHydra was decommissioned on November 14, 2018. Thelogin nodes and the disk system were available for twomore weeks until end of November 2018 so that the userscould save their data. Since then, the GPFS file systems/hydra/u and /hydra/ptmp are no longer accessible onthe new HPC cluster Cobra and on the Draco cluster.

Using the hyperthreading feature on theCobra cluster

With the update of the HPC system Cobra to Omni-Path/libfabric software version 10.8 in October 2018, theruntime behaviour of MPI has changed such that over-subscription of physical cores ("hyperthreading") withMPI tasks is now disabled by default. In order tomake use of hyperthreading by running more than 40MPI tasks per node on Cobra, the environment variablePSM2_MULTI_EP has to be set to 0 (see the corre-sponding example batch scripts on the Cobra webpage).Hybrid MPI/OpenMP jobs which utilize hyperthreadingby using multiple threads (and hence require at most 40MPI tasks per node) are not affected by the change.

NEC SX-Aurora Development Platform

The MPCDF is operating an NEC SX-Aurora TSU-BASA test system (variant A300-2) with two so-calledVector Engines (VE). Each Vector Engine comprises8 compute cores and 48 GB of high-bandwidth mem-ory (HBM2). In a very similar manner to a clas-sical vector architecture, the cores implement a 256-

word vector-processing pipeline, and 3 floating multiply-add (FMA) units. Thanks to the high memory band-width of 1 TByte/s per VE (reachable with 4 threads orcores, respectively), the peak floating-point (Flop) perfor-mance of 2450 GFlop/s (double precision) can already beachieved at an arithmetic intensity of 2.5 Flop/byte whichmakes SX-Aurora an attractive platform in particular formemory-bandwidth limited applications. For comparison:An Intel Skylake node (MPCDF Cobra with 2x Intel Xeon6148) with a memory bandwidth of approx. 200 GB/sand a peak floating-point performance of 3000 GFlop/srequires an arithmetic intensity of at least 15 Flop/byte.However, compared with common cache-based SIMD ar-chitectures (e. g. Intel Xeon or AMD Epyc) high floating-point performance can only be achieved by data-parallelprocessing of comparably long "vectors" (i. e. arrays withat least 256 elements) in a pipeline. More details aboutthe architecture can be found at the respective NEC web-page.The Vector Engines are realized as PCIe cards that canbe installed in a standard CPU server. From the pointof view of application development, this is implementedin a transparent manner, i. e. unlike a GPU a Vector En-gine presents itself to the application like a standard CPUnode with 8 cores and 48 GB shared memory runningLinux. The application runs entirely on the Vector Engine,and the host CPU is "invisible" to the application and isonly used for I/O and network communication. Thus,unlike a GPU, SX-Aurora is not an accelerator architec-ture and hence does not require special "offload" pro-gramming models (such as CUDA, OpenACC, or alike).The application programming is based on the well-knownlanguages (C, C++, Fortran) and parallelization mod-els (MPI and/or OpenMP within a Vector Engine, MPIfor communication between the Vector Engines). NECprovides corresponding compilers, MPI and numerical li-braries (BLAS, LAPACK, ScaLAPACK, FFT, ...), as wellas analysis tools.The system is open to application developers of the MaxPlanck Society (open a ticket in the MPCDF helpdeskto gain access). Interested users are invited to performporting and evaluation studies in collaboration with theMPCDF applications group.

Bits & Bytes, No. 200, January 2019 3

Data-Analytics on MPCDF HPC systems

Andreas Marek, Giuseppe Di Bernardo, John Alan Kennedy, Luka Stanisic

In the recent past the demand to use HPC systems for au-tomatically processing and analyzing large amounts of sci-entific data has grown significantly. Specifically, in manyscientific domains, there is rapidly growing interest in ap-plying analysis methods from the field of machine learn-ing, especially the so-called deep learning with artificialneural networks (ANN), an approach which is extremelypowerful, but is not easily portable to HPC systems.In order to address this trend and the future demands ofour users, a new data-analytics team has been formed tosupport the scientists with their data-intensive projectson the HPC systems of the MPCDF. Being tightly linkedwith the well-established application and data groups ofthe MPCDF, the data-analytics team will assist the usersin deploying their data-analysis projects in an efficient andscalable way on current and future HPC systems. Thesupport ranges from providing optimized variants of soft-ware frameworks to the implementation and paralleliza-tion of algorithms and software.

Parallel data analysis

Apache SparkApache Spark is an open-source cluster-computing frame-work that supports big-data and machine-learning appli-cations in a scalable way. Apache Spark provides a genericdata-analytics platform with many plugins and libraries,it offers the possibility to parallelize many kinds of data-analysis software and additionally provides support formachine-learning algorithms.The MPCDF provides the ability to run a Spark applica-tion on the HPC systems Draco and Cobra via the Slurmbatch system. For details and example submission scriptswe refer the user to a previous Bits & Bytes article "Sparkon Draco" or the webpage about data-analytics software.

Machine-learning frameworks

The MPCDF provides support for several data-analyticsand machine-learning frameworks on the HPC systemsDraco and Cobra. In the following section we will in-troduce the software which is currently supported andprovide basic information to get you started.

1. Scikit-learnScikit-learn is a Python-based package which provides nu-merous tools and algorithms for machine learning. Exten-sive documentation about the software package itself canbe found at the http://scikit-learn.org webpage. Scikit-learn is provided on the HPC systems Draco and Cobraand can be loaded via the commands

$ module load anaconda /3/5.1$ module load scikit -learn /0.19.1

Please note that although our installation makes effi-cient use of a single node of an HPC system, scikit-learndoes not natively support the parallelization of machine-learning tasks across multiple nodes. It is, however,well suited for proto-typing and testing different machine-learning algorithms. We advise users to test such algo-rithms on the interactive Slurm partitions and not on thelogin nodes of the HPC systems.

2. TensorflowTensorflow is the de-facto standard framework formachine-learning, especially deep-learning applications.On the HPC systems Draco and Cobra we provide a highlyoptimized version of Tensorflow. It is possible to useTensorflow directly via the Python Tensorflow API (doc-umentation is available at https://www.tensorflow.org)or to use Tensorflow as a backend of the higher-levelKeras Python package (documentation is available athttps://keras.io/). The Tensorflow installation at theMPCDF provides functionality for training and/or infer-ence of a neural network on a single node of the HPCsystems. The parallelization of the training over multi-ple nodes is supported via the Horovod (see Section 3)extension to Tensorflow.

2.1 Tensorflow/GPUTensorflow achieves its best performance when usingGPUs, this is due to the large amount of matrix-matrixmultiplications used internally. Thus, if the Tensorflowcomputations fit into the memory of a GPU (8 GB onDraco, 32 GB on Cobra), it is desirable to use the GPUversion we provide.To use the GPU version of Tensorflow 1.12 on Draco orCobra load the software as follows:

# on DRACO$ module load gcc /6.3.0# on COBRA$ module load gcc/6

$ module load cuda$ module load anaconda /3/5.1# for cuda /9.1 (DRACO)$ module load cudnn /7.0$ module load nccl /2.2.13# for cuda /10.0 (COBRA)$ module load cudnn /7.4$ module load nccl /2.3.7

$ module load tensorflow/gpu /1.12.0

If Keras is required, the following extra command isneeded:

Bits & Bytes, No. 200, January 2019 4

$ module load keras /2.2.4

For more details on how to use Tensorflow with GPUs onour HPC systems, please have a look at the data-analyticswebpage, where more exhaustive documentation and ex-amples are provided.

2.2 Tensorflow/CPUIn case a Tensorflow application needs more memory thanis available on the GPUs of the HPC systems, one can usethe CPU version, which we also provide on Draco and Co-bra. The CPU version allows applications to use all thememory of the node (between 96 GB and 768 GB, de-pending on the HPC system and the node). It should bepointed out that Tensorflow will run significantly slowerin the CPU version than in the GPU version.To use the CPU version of Tensorflow 1.12 load the soft-ware as follows:

$ module load gcc/8$ module load anaconda /3/5.1$ module load tensorflow/cpu /1.12.0

If Keras is required, the following extra command isneeded:

$ module load keras /2.2.4

3. Tensorflow-HorovodHorovod is an extension to Tensorflow which allows neu-ral networks to be trained in parallel on multiple nodes ofour HPC systems with the "data-parallelism" approach.A detailed description on how to use Horovod is beyondthe scope of this article. However, we wish to mention

here that to use Horovod one has to modify the plainTensorflow (or Keras) Python code and adapt the Slurmsubmission scripts for parallel computations.Examples for both steps can be found at the data-analytics webpage. Once the modifications are done, thetraining of a neural network can be sped up significantly,as we show in Figure 3:

Figure 3: Strong scaling of the training of the VGG-16network on several nodes of the HPC system Cobra.

4. Apache SparkIn addition to providing a generic framework for data an-alytics Apache Spark also supports machine learning vianative libraries (in the form of MLlib) and techniques existto deploy distributed machine-learning and deep-learningsolutions (including Tensorflow models) via Spark.

Keeper - Archive the way you work

Raphael Ritz

With the Keeper Service you can easily sync and shareyour project data. In addition, Keeper ensures thatyour work is properly archived in the Max Planck infra-structure. Convince yourself of the reliable and high-performance file-syncing algorithms and the built-in fileencryption. Support your team with the Keeper collab-oration functionalities. By storing your data in Keeperyou can be sure that you fulfill the archiving regulationsof the Max Planck Society as well as those of the Ger-man Research Foundation to ensure "good scientific prac-tice". Additionally you may deposit your data into theKeeper Service for long-term compliant archiving after

your project ends.The Keeper Service is free of charge for all Max Planckemployees. If you are from an external organization andcollaborate with the Max Planck Society you are also in-vited to exploit the Keeper Service for all Max Planckrelated projects.Keeper is offered by the Max Planck Digital Library(MPDL) and operated by the MPCDF. It is implementedon top of the cloud-storage solution Seafile. For furtherinformation please turn to keeper.mpdl.mpg.de or contactthe Keeper support team at keeper@mpdl.mpg.de.

Bits & Bytes, No. 200, January 2019 5

New Projects and Collaborations

Hermann Lederer

Enabling of Plasma Physics SimulationCodes towards Exascale

In the context of the new PRACE-6IP project as of April2019, PRACE will be co-funding the enabling of simu-lation codes towards Exascale in various scientific disci-plines. A call for proposals with a deadline in August 2018resulted in 15 proposals. A consortium of the Finnish Cen-tre for Science and Technology (CSC), the Max PlanckComputing and Data Facility and the University of Ljubl-jana contributed the proposal "Modernisation of PlasmaPhysics Simulation Codes for Heterogeneous Exascale Ar-chitectures", with science community support by the MaxPlanck Institute for Plasma Physics for the GENE micro-turbulence simulation code, the University of Helsinki forthe VLASIATOR simulation code, and the Aalto Univer-sity for the ELMFIRE code. This proposal received thehighest ranking and will be one out of seven such projectswith PRACE funding starting in April 2019 with a dura-

tion of 30 months.

Munich School for Data Science

Several key science institutions in the Munich area col-laborate to address the urgent need for the training ofnext generation scientists at the interface of data sci-ence and various science domains, by creating the MunichSchool for Data Science (MuDS) as a part of the MunichAlliance for Computation and Data. MuDS integratesthe Helmholtz Centres in the Munich area (HMGU, IPP,DLR) with the Technical University of Munich (TUM),the Ludwig-Maximilians-Universität München (LMU) andtwo major computing and data facilities, the Leibniz Su-percomputing Centre (LRZ) and the Max Planck Com-puting & Data Facility (MPCDF). For further detailsplease see the press release published by the HelmholtzZentrum München.

NewsHermann Lederer, Markus Rampp

Data stored in HPSS exceeds 100 PB

In 2011 the RZG introduced HPSS as the major Hierar-chical Storage Management (HSM) system. HPSS hasbeen developed by a consortium of major Departmentof Energy sites in the US and is maintained by IBM.The introduction of HPSS was triggered by the increaseddemands for disk-to-tape bandwidth through the sup-port of multiple streams. Recently, the amount of datastored in HPSS has surpassed the 100 PB threshold, andin the list of publicly disclosed HPSS deployments, aspublished in http://www.hpss-collaboration.org/customersT.shtml, the MPCDF belongs to the top10 scientific data centers worldwide; more precisely, theMPCDF has been ranked as no. 8 and is the first entryfor Germany.

IAEA challenge award for IPP/MPCDFteam

In 2018 the International Atomic Energy Agency IAEApublished a challenge for the analysis of material defectsfrom molecular dynamics simulations, the "IAEA Chal-lenge on Materials for Fusion". For the competition,the IAEA provided datasets which described the positionsof all tungsten or iron atoms 40 ps after a fast parti-cle impacting the originally perfect metal crystal. Fromthe large datasets of several million atom coordinates persimulation the changes in the crystal induced by the im-pact had to be identified. A team of scientists from theMax Planck Institute for Plasma Physics (IPP) and theMPCDF was awarded with the first prize (5000 EUR) outof many proposals from all over the world, and was in-vited to present their results at the IAEA headquarters inVienna. The methodology and developed software is de-scribed at the IAEA webpage and can be made availableby the authors on request.

Bits & Bytes, No. 200, January 2019 6

EventsKlaus Reuter, Sebastian Ohlmann, Raphael Ritz, Hermann Lederer

Course on "Python for HPC"

In November 2018, a two-day course on "Python forHPC" was taught by MPCDF staff at Garching. Morethan 70 participants attended the lectures and hands-onexercises on NumPy, Cython, Numba, multiprocessing,mpi4py, visualization, and other Python-related topics.The event was motivated by the fact that the use ofPython-based codes on HPC systems has increased sig-nificantly over the last couple of years, and at the sametime it is highly critical to apply appropriate programmingtechiques to avoid pitfalls and achieve good performance.A second issue of the course is planned to take place inspring 2019 and will be announced soon.

RDA Deutschland Tagung 2019

This year’s meeting of the German-speaking part of theResearch Data Alliance – co-organized by the MPCDFthrough its engagement in the RDA-Europe project – willtake place in Potsdam on February 19 and 20 at the Geo-ForschungsZentrum (GFZ). The meeting is open to any-

one with an interest in advancing research data manage-ment. In addition to general updates on RDA activitiesthe focus of this year’s meeting will be the forthcomingnational research data infrastructure in Germany and itsembedding into the international research scene. Furtherdetails are available from the meeting site.

International HPC Summer School 2019

The International HPC Summer School 2019 is goingto take place from July 7 to 12 at the campus of theRIKEN Center for Computational Science (R-CCS) inKobe, Japan. Graduate students and postdoctoral schol-ars from institutions in Canada, Europe, Japan and theUnited States are invited to apply. The summer schoolis organized by XSEDE for the US, PRACE for Europe,R-CCS for Japan and the SciNet HPC Consortium forCanada. Interested students are invited to apply by Febru-ary 4, 2019. School fees, meals and housing will becovered for all accepted applicants, as well as interconti-nental flight costs. Further information and application:http://www.ihpcss.org