MANUFACTURINGWEATHER FORECASTING

SIMULATIONSON HPC INFRASTRUCTURES

LADISLAV HLUCHÝV. ŠIPKOVÁ, M. DOBRUCKÝ, J. BARTOK, B.M. NGUYEN

INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES

ECW 2016 - ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016

PARTNERS

• IISAS: INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES (ACADEMIC SECTOR)

• MICROSTEP-MIS: MONITORING AND INFORMATION SYSTEMS (COMMERCIAL SECTOR)

• IMS MODEL SUITE: COMPLEX SOFTWARE SYSTEM FOR METEOROLOGY AND CRISIS MANAGEMENT

• THIS PAPER PRESENTS A PART OF MANUFACTURING WRFON HPC INFRASTRUCTURE FOR IMS MODEL SUITE

WRF - WEATHER RESEARCH AND FORECASTING

• DESIGNED FOR RESEARCH AND OPERATIONAL PURPOSES

• NUMERICAL WEATHER PREDICTION

• ATMOSPHERIC SIMULATION

• TWO DYNAMIC SOLVERS

• ARW: ADVANCE RESEARCH WRF

• NMM: NON-HYDROSTATIC MESOSCALE MODEL

• FLEXIBLE AND PORTABLE CODE

• SEQUENTIAL

• PARALLEL (MPI) WITH OR WITHOUT MULTI-THREADING

• SUPPORTS A TWO-LEVEL DOMAIN DECOMPOSITION

• AT FIRST INTO PATCHES FOR DISTRIBUTED MEMORY,

• THEN WITHIN EACH PATCH MULTI-THREADING IS APPLIED FOR SHARED MEMORY

OBJECTIVES

• DEVELOPMENT OF MANAGEMENT TOOLS TO FACILITATE THE EXECUTION OF THE WRF SIMULATION PROCESS ON HPC INFRASTRUCTURES• LOCAL HPC CLUSTER

• GRID INFRASTRUCTURE (EGI)

• PERFORMANCE INVESTIGATION OF PARALLEL WRF MODELS TO FIND OUT THE MOST SUITABLE CONFIGURATION WITH THE GIVEN INPUT SCENARIO FOR 3D METEOROLOGICAL MODELLING• MPI

• MPI + OPENMP

• THE NUMBER OF COMPUTE NODES, CORES, MPI PROCESSES, OPENMP THREADS

• THE MANAGEMENT TOOLS ARE ALSO USED FOR PARAMETER TUNING OF THE MODELS (FOR IMS BY MICROSTEP-MIS) THAT REQUIRES • TENS OF EVALUATIONS OF THE PARAMETERIZED MODEL ACCURACY

• EACH EVALUATION OF THE MODEL PARAMETERS REQUIRES RE-RUNNING OF THE HUNDREDS OF METEOROLOGICAL SITUATIONS COLLECTED OVER THE YEARS AND COMPARISON OF THE MODEL OUTPUT WITH THE OBSERVED DATA

3D METEOROLOGICAL MODELLING• DOMAINS - WEATHER MODELLING

• HORIZONTAL, VERTICAL AND TIME RESOLUTION,

• SO THE MODEL CAN CATCH LOCAL CONDITIONS

• METEOROLOGICAL INITIAL AND BOUNDARY CONDITIONS

• FROM THE GLOBAL MODEL GFS (GLOBAL FORECASTING SYSTEM) OF US NATIONAL WEATHER SERVICE

• THE SETTING ENABLED TO MODEL THE ARABIAN PENINSULA WEATHER

• THE UPPERMOST DOMAIN WITH THE RESOLUTION 50X50 KM

• THE FINAL DOMAIN WITH THE RESOLUTION 1.8 KM, AROUND DUBAI AND ABU DHABI

WRF SIMULATION

• WRF SIMULATION CONSISTS OF MANY EXECUTABLE PROGRAMS

• VARIOUS TYPE AND COMPLEXITY, SEQUENTIAL AND PARALLEL

• TAKING A DIFFERENT NUMBER OF PROCESSOR CORES FOR EXECUTION

• WRF WORKFLOW - DAG GRAPH• (JOB 1) WPS PREPROCESSING,

• (JOB 2) WRF MODELING,

• (JOB 3) UPP POST-PROCESSING

Pi – MPI process Tj – OpenMP

thread

WRF WORKFLOW – MORE DETAILS• JOB 1 - WPS PREPROCESSING: CONVERSION OF INPUTS FROM GRIB TO NETCDF

FORMAT USING

• GEOGRID.EXE (SERIAL/MPI)

• UNGRIB.EXE (SERIAL)

• METGRID.EXE (SERIAL/MPI)

• JOB 2 - WRF MODELING - NUMERICAL MODELING USING

• REAL.EXE – INITIALIZATION - REAL DATA PREPROCESSOR (MPI/MPI+OPENMP)

• WRF.EXE – NUMERICAL INTEGRATION - ARW SOLVER (MPI/MPI+OPENMP)

• JOB 3 - UPP POST-PROCESSING

• CONVERSION OF OUTPUTS FROM NETCDF TO GRIB FORMAT USING UNIPOST.EXE(SERIAL/MPI) IN A NESTED CYCLE FOR ALL HOURS OF THE PREDICTED TIME PERIOD

• THERE IS NO DEPENDENCY BETWEEN PROCESSING DATA OF INDIVIDUAL HOURS, SO, THE JOB CAN BE STRUCTURED AS A PARAMETRIC STUDY (PS), WHERE EACH SUB-JOB HANDLES A SECTION OF THE TIME PERIOD

WRF WORKFLOW EXECUTION

• STARTS ON THE UI MACHINE THROUGH THE INVOCATION OF THE WRF WORKFLOW-MANAGER ENCOMPASSED WITH NEEDED INPUT PARAMETERS

• IS REALIZED WITHIN THE “RUNNING-ENVIRONMENT” LOCATED IN THE SHARED ADDRESS SPACE WHICH HAS THE DIRECTORY STRUCTURE• GEOG – GEOGRAPHICAL DATA, SEVERAL GEO-TABLES

• CFG – CONFIGURATION FILES FOR INPUT SCENARIO AND SIMULATION OPTIONS

• PARM – UPP POST-PROCESSING PARAMETERS

• BIN – RUN-SCRIPTS AND EXECUTABLES

• INPUT_ARCH – INPUT DATA FILES

• OUTPUT_ARCH – OUTPUT DATA FILES

• WPS_RUN – WPS PREPROCESSING

• MODEL_RUN – WRF MODELING

• POSTPR_RUN – UPP POST-PROCESSING

IISAS HPC CLUSTER

v HARDWARE CONFIGURATION

52X IBM DX360 M3 (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB SCRATCH DISK), 2X IBM DX360 M3 (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB SCRATCH DISK, NVIDIA TESLA M2070: 6 GB RAM + 448 CUDA CORES), 2X X3650 M3 MANAGING SERVERS (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 6X 500 GB DISKS), 4X X3650 M3 DATA-MANAGING SERVERS (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB DISKS, 2X 8 GBPS FC), 1X X3550 M4 SERVER (1X INTEL E5-2640 @2.5GHZ, 8 GB RAM, 2X 500 GB DISKS), INFINIBAND 2X 40 GBPS (IN 52+2+2+4 NODES), 2X DS3512 WITH 72TB DISKS

v SOFTWARE INSTALATION

• WRF PACKAGE VERSION 3.7.1 (WRF, WPS, TERRESTRIAL DATASETS),

• UPP VERSION 3.0,

• LIBRARIES NETCDF 4, JASPER 1.7,

• GNU COMPILERS VERSION 4.4.7 (GFORTRAN, GCC, OPENMP LIBRARY),

• OPEN MPI VERSION 1.10.0

PERFORMANCE RESULTS WRF MODEL: SEQUENTIAL ON THE LOCAL CLUSTER

• PREDICTION TIME PERIOD

• 3 HOURS IN THIS PAPER FOR SCALING WRF SIMULATIONS FOR TESTING PURPOSE WITH GIVEN HW/SW CONFIGURATIONS

• 48 HOURS IN REAL SIMULATIONS (MICROSTEP-MIS) TO MODEL THE ARABIAN PENINSULA WEATHER

• THE NEED OF HPC TO ACCELERATE SIMULATIONS

Number ofnodes

Number ofcores per

node

Execution time

hh:mm:ssWPS 1 1 00:39:54WRF 1 1 15:57:53UPP (2 jobs) 1 1 00:03:48Complete simulation process 16:41:35

PERFORMANCE RESULTSWRF MODEL: MPI ON LOCAL CLUSTERFIXED NUMBER OF CORES PER NODE

Number of

nodes

Number of

cores per node

Number of

MPI processes

Execution time

hh:mm:ss

WPS 1 10 10 00:04:22WRF 1 8 8 02:36:33WRF 2 8 16 01:27:01WRF 4 8 32 00:49:03WRF 8 8 64 00:30:13WRF 16 8 128 00:20:47WRF 32 8 256 00:13:57UPP (2 jobs) 1 3 3 00:01:44Complete simulation process (best) 00:20:03

PERFORMANCE RESULTSWRF MODEL: MPI + OPENMP ON LOCAL CLUSTER

FIXED MPI PROCESSES

Number ofnodes x cores

Number ofMPI

processes( per node)

Number ofOpenMPthreads

Execution time

hh:mm:ss

WRF 8x12 32 (4) 2 00:31:31

WRF 16x12 32 (2) 4 00:20:52

WRF 16x12 32 (2) 6 00:17:21

WRF 32x12 32 (1) 8 00:15:47

WRF 32x12 32 (1) 10 00:15:15

WRF 32x12 32 (1) 12 00:15:20

PERFORMANCE RESULTSWRF MODEL: MPI + OPENMP ON LOCAL CLUSTER

FIXED NUMBER OF OPENMP THREADS

Number ofnodes x cores

Number ofMPI

processes(per node)

Number ofOpenMP threads

Execution time

hh:mm:ss

WRF 8x12 32 (4) 3 00:24:44

WRF 12x12 48 (4) 3 00:19:28

WRF 16x12 64 (4) 3 00:17:27

WRF 24x12 96 (4) 3 00:13:49

WRF 32x12 128 (4) 3 00:12:24

WRF 16x12 32 (2) 6 00:17:21

WRF 24x12 48 (2) 6 00:14:31

WRF 32x12 64 (2) 6 00:12:09

WRF 40x12 80 (2) 6 00:12:01

WRF MODELMPI ON GRID INFRASTRUCTURE EGI

• WRF RUNNING-ENVIRONMENT IN ITS INITIAL STATE, ALL EXECUTABLES AND INPUT FILES ARE STORED IN GRID STORAGE ELEMENT (SE), FROM WHICH THEY ARE DOWNLOADED

• GEOGRAPHICAL DATASETS (174 GB) ARE LOCATED IN CLUSTER SHARED ADDRESS SPACE, THEY DO NOT PARTICIPATE ON THE DATA TRANSFER

• GRID WRF WORKFLOW IS DESIGNED AS ONE GRID JOB ENCAPSULATING ALL TASKS: WPS+WRF+UPP

• MPI PROGRAMS ARE EXECUTED USING MPI-START

• OUTPUT OF SIMULATION IS UPLOADED TO STORAGE ELEMENT (SE)

• TIME OVERHEAD BY DATA TRANSFERS BETWEEN CE AND SE: 2 MINUTES

Grid UI – Grid User InterfaceWMS – Workload Management System VO – Virtual OrganizationCE – Computing ElementsGG – Grid GateLRMS – Local Resource Management SystemWN – Working NodeSE – Storage Element

PBS – Portable Batch System

CONCLUSION

• MANAGEMENT TOOLS ARE BUILT AND FULFILL DESIGNED PURPOSES

• TO LOCATE THE OPTIMAL CONFIGURATION WITH GIVEN SCENARIO

• FOR IMS MODEL PARAMETER TUNING (MICROSTEP-MIS)

• HYBRID PROGRAMMING MODEL (MPI + OPENMP) SEEMS A NATURAL FIT FOR THE WAY MOST CLUSTERS ARE BUILT TODAY

• THE GRID OVERHEAD IS CAUSED MAINLY BY THE TRANSFER OF BIG FILES BETWEEN THE SE AND CE

FUTURE DIRECTIONS

ØGRID

• AT THE MOMENT, IN EUROPEAN GRID INFRASTRUCTURE (EGI), ONLY A FEW GRID SITES AND VIRTUAL ORGANIZATIONS (VO) ARE SUPPORTING MPI ANDOPENMP APPLICATIONS

ØCLOUD

• PERFORMANCE OVERHEAD ASSOCIATED WITH VIRTUALIZATION OF INTERCONNECTION NETWORK

• WRF IS REPORTED TO RUN ON VIRTUALIZED INFINIBAND INTERCONNECT WITH ONLY 15% OVERHEAD WHICH MAKES FULLY VIRTUALIZED HPC CLUSTERS VIABLE SOLUTION

ØACCELERATORS

• PARTS OF WRF WERE PORTED TO NVIDIA GPU AND INTEL XEON PHI WITH PROMISING RESULTS

THANK YOU FOR YOUR ATTENTION

MANUFACTURING WEATHER FORECASTING SIMULATIONS ON HPC INFRASTRUCTURES

LADISLAV.HLUCHY@SAVBA.SK

INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES

ECW 2016 - ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016