Introduction to Advanced Research Computing (ARC)

September 28, 2017

1

By: Pier-Luc St-Onge

Financial Partners

2

Setup for the workshop1. Get a user ID and password paper (provided in class):

##:**********

2. Access to local computer (replace “##” and “___” with appropriate values, “___” is provided in class):a. User name: csuser## (ex.: csuser99)b. Password: ___@[S## (ex.: sec@[S99)

3. Slides:

http://tinyurl.com/cq-intro-arc-20170928

3

Outline

● Introduction to ARC○ Laptop vs HPC Cluster

● Moving Data○ With Globus

● Software Development○ Types of Parallelism○ Scientific Software and Modules○ Best Practices

● The Scheduler and Job Submission● Compute Canada and Calcul Québec

4

5

Introduction to ARC

Introduction

● This workshop is built from the HPCS 2015 - Introduction to ARC workshop:○ Etherpad with extra information:

■ https://etherpad.mozilla.org/hpcs2015-intro-arc

● About Post-its○ Blue-green : OK, exercise done○ Orange-red : not OK, I have a question or a problem

● Presentation:○ Instructors and participants○ Name and research domain

6

Other research domains

● Genomics● Molecular Dynamics (MD)● Finite Element Analysis (FEA)● Computational Fluid Dynamics (CFD)● Astronomy and Astrophysics● Geospatial Data Analysis● 3D renderings, Image Analysis● Artificial Intelligence / Deep Learning (AI)● Humanities and Social Sciences (DH)

7

Advanced Research Computing

● High Performance Computing (HPC)○ Parallel code○ Profiling and optimizing code○ Running code on accelerators

● Management of massive data● Complete “pipelines” for jobs submission● Sharing data● Web portals for special projects● Cloud computing / storage

8

ARC is also About Having Enough Resources● https://www.socrative.com/

○ Student login - room PIERLUCSTONGE● Problem: we want to compute a dense matrix

multiplication C = A * B○ Each square matrix is made of ~1 billion floating

point numbers in double precision● How much memory do you need for this job?

a) 3.2 GBb) 4.3 GBc) 8.6 GBd) 12.9 GBe) 25.8 GB

9

Different Tools for Different NeedsPortable computer

● 2-4 cores● 4-16 GB● Local storage, USB

○ 128 GB to 4 TB● Wi-Fi and Ethernet

○ General usage○ Cloud storage

● Gaming GPU○ 32 bits

● Direct access

10

HPC Compute node

● 8-64 cores● 16-3072 GB● Network storage

○ 150 TB to 16 PB● Ethernet and fast

network (IB, OPA)○ Multi-node tasks

● Accelerators○ 64 bits, ECC mem.

● Access by queued jobs

Zoom on a HPC Compute Node

11

Local hard-drive (scratch)

Proc. Proc.

Proc.Proc.

RAM

RAM

RAM

RAM

RAM

RAM

RAM

RAM

Proc. Proc.

Proc.Proc.

Infiniband/Omni-Path

Ethernet

Accelerator

Why Infiniband and Omni-Path?

12

● Some tasks need more than one compute node○ Split workload, memory usage and storage access

● High bandwidth○ Ethernet: 1 Gbps (~100 MB/s)○ Infiniband and Omni-Path: 20-100 Gbps (2-12 GB/s)

■ Best for I/O on large files

● Reduce latency○ Ethernet: ~30µs (33k iops = I/O operations per sec.)○ IB and OPA : 2 µs or less (500k iops or more)

■ Best for several small communications

Storage Available on Compute Nodes

13

Project spaceScratch space

Bandwidth

Latency

Local scratch

RAMDISK

Capacity

(Not to scale)

Typical Cluster

14

Ethernet &

IB or OPA Login node

ssh me@...

InternetStorage

homeprojet

scratch

Scheduler Nodes

Job Job

Typical Cluster Usage

15

1. Connect to a login nodea. Import datab. Compile code or install applications

2. Submit jobs to the schedulera. Define needed resources: processors, accelerators,

memory, walltime limitb. Wait in queue - monitor your positionc. Automatic execution of your jobs on compute

nodes

3. Get your results

Typical Environment on Cluster● The operating system is usually Linux (CentOS,

Scientific Linux, …)● To connect to a cluster, we usually use an encrypted

session through SSH (Secure SHell)○ To transfer data safely: SCP (Secure CoPy) and

others tools…○ Some web portals may also provide an access

● Development environment○ Multiple text editors○ Scientific software○ Version control software

● Tools for transferring large amount of data○ 10 Gbps, 40 Gbps or 100 Gbps between sites

16

Moving Data

With Globus

17

Moving Data Over SSH

● Secure CoPy with SCP or WinSCP:○ Most clusters only allow SSH connections

■ SSH: Secure SHell (username and password)○ SCP uses the SSH protocol to copy files

■ Simple tool for small amount of data (<1GB)○ On Windows: use WinSCP, FileZilla or MobaXterm

■ Two panels, where one side shows your files on the cluster, then drag-and-drop files or folders

○ On Linux or Mac OS X: use the scp command:■ scp file.txt user@cluster:folder/■ scp -pr user@cluster:folder ./

18

Globus - Why?

● Secure CoPy with SCP or WinSCP:○ One process and one thread○ Fails on dropped connections

● BBCP - parallel transfers○ Need a calibration for the round-trip time○ Command line with multiple arguments

● Globus - Web interface and CLI○ No need for calibration○ Convenient Web interface○ Confirmation email after the transfer○ Possible to share data with other users

19

Go to Compute Canada Research Portal● CC > Research Portal > National Services > Data Movement (Globus)

https://www.computecanada.ca/research-portal/globus-portal/

20

Create an account if you do not have

OrLog in to your account if you already have an account

Account Creation and Login1. Go to: https://globus.computecanada.ca/SignIn#step=signup2. Select Compute Canada if you have a CCDB account, or Globus ID if

not. To create a Globus ID account: https://www.globusid.org/create

21

Compute Canada Login and Globus ID Account Creation

22

Exercise: Sign-up!

23

● https://globus.computecanada.ca/SignIn#step=signup● Globus ID credentials are stored on Globus:

○ Can be a different username than the one on CCDB, or it can be the same. But both accounts (on CCDB and Globus ID) are different!

○ For sure, use a different password!

● Put the green Post-it on the computer screen when done

Globus Home Interface

24

To Logout

Globus Endpoint on your

computerTransfer options

Endpoint Viewports

Choosing an Endpoint at Compute Canada

25

Authentication to an Endpoint

26

● The authentication is done on a server corresponding to the endpoint○ Globus will never know your credentials for a given

cluster/endpoint○ The authentication server only grants access to

Globus.

Files and Directories

27

Exercise: Import the Material

● Connect to computecanada#guillimin○ Use the class account credentials (on your paper)

■ class##■ **********

● Connect to intro-arc_guillimin_plstonge○ Select both exercises and pictures folders○ Click on the blue button to initiate the transfer to

your (class##) home directory

● Wait for the email report○ When the transfer is completed, put the green

Post-it on top of the computer screen28

Confirmation Email

29

TASK DETAILS

Task ID: 5ef1f2c6-81f5-11e6-afd7-22000b92c261

Task Type: TRANSFER

Status: SUCCEEDED

Source: intro-arc_guillimin_plstonge (...)

Destination: computecanada#guillimin (...)

Label: n/a

https://www.globus.org/app/activity/...

Transfer Report

30

Managing Endpoints

31

Create a Local Endpoint

32

Create a Local Endpoint

33

1. Choose a unique Endpoint namea. Generate the Setup Keyb. Copy the Setup Key in a text editor

2. Download and Install Globus Connect Personala. Start it and paste the Setup Key

3. A Globus icon ( g) appears in the taskbara. [Right-]click -> Preferences -> Accessb. Select accessible directories you want to export

4. Use your Endpoint: user#uniqueName

Sharing Option

34

Software Development

Types of Parallelism

35

Vocabulary

● Serial tasks○ Any task that cannot be split in two simultaneous

sequences of actions○ Examples: starting a process, reading a file, any

communication between two processes

● Parallel tasks○ Data parallelism: same action applied on different

data. Could be serial tasks done in parallel.○ Process parallelism: one action on one set of data.

Action split in multiple processes or threads.■ Data partitioning: rectangles or blocks

36

Parallel tasks

● Shared memory○ Multiple threads share the same memory space in a

single process: full read and write access.

● Distributed memory○ Each process has its own memory space○ Information is sent and received by messages

● Parallel efficiency (scaling)○ Amdahl’s law: how long does it take to compute a

task with an infinite number of processors?○ Gustafson's law: what size of problem can we

solve in a given time with N processors?

37

Software Development

38

Scientific Software and Modules

Connecting to Guillimin - SSH Client

● From Linux, open a Terminal window● From Mac OS X, open a Terminal window● From Windows, install PuTTY, MobaXterm

or any other SSH (2.0 compliant) client○ http://www.chiark.greenend.org.uk/~sgtatham/putty

/download.html■ putty.exe is a standalone executable■ Simply run putty.exe

○ http://mobaxterm.mobatek.net/

39

Connecting to Guillimin - Terminal

● Server name: guillimin.calculquebec.ca● Username: class##

○ ## = Two digits on the first line of the small piece of paper

● Password: **********○ Sequence of 10 characters on small piece of paper

● Linux/Mac OS X example:ssh class99@guillimin.calculquebec.ca

○ Accept the server’s key fingerprint

40

Connecting to Guillimin - PuTTY

41

Scientific Software

● You will use or create scientific software○ Applications○ Libraries

● Make sure you are using a code language widely used in your research domain

● Make sure your code is well documented and easy to maintain○ Keep in mind that future researchers are going to

use your software

● Do not reinvent the wheel: use modules!42

Modules

● Exercise:module avail

● Compilers: GNU, Intel, LLVM, PGI, Java● Language interpreters: Python, R● Libraries and other applications:

○ OpenMPI, MVAPICH2, Boost○ MKL, FFTW○ GROMACS, NAMD, Quantum-Espresso, NetCDF

● Other tools:○ Paraview, GNU-Parallel, GDB, Valgrind

43

Exercises with Moduleswhich gcc

module add gcc/4.8.2

module list

which gcc

module rm gcc/4.8.2

module add ifort_icc openmpi

echo $LD_LIBRARY_PATH

echo $CC

module purge

44

Lmod/EasyBuild-based hierarchical module system● A default set of modules on Cedar and Graham● Some legacy and EasyBuild modules are available

on Guillimin. To look for any R: module spider R

Exercise on Guillimin:1. Load the main toolchain Intel + OpenMPI + MKL:

module load iomkl/2015b

2. List loaded modules:module list

3. You now have access to much more modules:module avail

45

Exercises withCompute Canada Modulesbash

source /software/soft.computecanada.ca.sh

module list

module avail

module load boost

module purge

exit

46

Introduction to GNU-Parallel● https://wiki.calculquebec.ca/w/GNU_parallel/en● Automatic online process scheduler for serial tasks

○ User provides a command line template and one or multiple sequences of arguments

● For N cores on a compute node, GNU-Parallel will start N processes○ As soon as one process is complete, it will start the

next process according to values in sequences● Example:

module load parallel/20150822

parallel echo {1} ::: $(seq 1 960)

47

Introduction to Threads

● Threads are used for parallel tasks on a single node

● One process, one shared memory space, but multiple threads

● Synchronization issues, like race conditions○ Two threads reading or writing to the same variable

● Instead of POSIX threads:○ OpenMP○ Threading Building Blocks

48

Introduction to MPI

● Message Passing Interface (MPI) is a standard for passing messages between multiple processes on multiple nodes○ Each process can know its rank number

● Same set of functions, but different implementations○ MPICH2, MVAPICH2, OpenMPI, Intel MPI,...

● Example:module load iomkl/2015b

mpiexec -n 4 date +%s.%N

49

Software Development

Best Practices

50

Version Control &Revision Control

● Bad practice:○ Keeping multiple versions in sub-directories○ Sending a specific version by email○ Trying to merge two versions manually

● Good practice: using a version control system○ Choose one that fits your project:

■ Git (git)■ Mercurial (hg)■ Subversion (svn)

○ Repository: shared on a remote server

51

Exercises with gitmkdir -p ~/projectA; cd ~/projectA

module add gitgit init # Local repos.

touch script.sh; git statusgit add script.sh # Ready to commit

git commit -m "intro-ARC" # Do commit

Add line “env > env.log” in script.shgit diff # See differencesgit add script.sh # Ready to commitgit commit -m "Variables" # Do commit

git log # Show past commits

52

One Last Good Practice -Using SSH Keys

● Having access with a local passphrase:ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (~/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

● Copy the public key to a remote site (id_rsa.pub):

scp .ssh/id_rsa.pub <user>@<server>:

ssh <user>@<server> \

"cat id_rsa.pub >> .ssh/authorized_keys"

53

Exercise - Code Compilation

54

● Go to the exercises folder

cd ~/exercises

● Load the following modules:

iomkl/2015b

Boost/1.59.0-Python-2.7.10

ImageMagick/7.0.1-6

● Build the application:

make

The Scheduler and Job Submission

55

The Scheduler

● Jobs are not running on login nodes● You need to submit your jobs to the

scheduler:○ Resources you need: n processors, memory,

accelerators○ Job duration○ Account name○ The script to run on worker nodes○ Where you want your results

https://wiki.calculquebec.ca/w/Running_jobs

56

Schedulers in Calcul Québec

57

Job component Torque + Maui/Moab

Submission commandTorque: qsub

Maui/Moab: msub

Account -A xyz-123-aa

Processors (small L)Some clusters force full nodes

-l nodes=N:ppn=P-l procs=PP

Memory (small L)-l pmem=1700m-l mem=20g

Job duration (small L) -l walltime=d:hh:mm:ss

Job name -N name

Where you want your results-o output_path-e error_path

your_app > out.log

Environment Variables During Job

58

Variable Description

$PBS_JOBIDJob IDEx.: 12345678.server.name

$PBS_JOBNAMEJob name set by -NEx.: introARC

$PBS_NODEFILEThe name of the file that contains a list of assigned nodes for the job

$PBS_O_WORKDIRThe current working directory ($PWD) when the job has been submitted.

$PBS_ARRAYIDOne of the values specified by the job array: -t 1-3,6,10-12

Demo:Edit and Submit a Job Script

59

● Go to projectA: cd ~/projectA

● Edit with: nano script.sh#!/bin/bash#PBS -l nodes=1:ppn=1#PBS -l walltime=5:00#PBS -N myEnvenv > env.log

● Submit job: qsub script.sh

Demo:Manage your Jobs

● Torque - jobs and job array detailsqstat -u $USERqstat -t -u $USER

● Maui / Moab - jobsshowq -u $USER

● (Optional) Torque - cancel jobqdel jobid

● (Optional) Maui / Moab - cancel jobmjobctl -c jobidcanceljob jobid

60

Scheduler on Cedar and Graham

● Slurm Workload Manager○ sbatch script.sh

■ #SBATCH --account=def-someuser■ #SBATCH --time=3:00 ## 3 minutes■ #SBATCH --nodes=2■ #SBATCH --ntasks-per-node=8■ #SBATCH --cpus-per-task=4■ #SBATCH --mem-per-cpu=3G

○ squeue -u $USER○ scontrol show job -dd JOBID○ scancel JOBID

61

Exercises:Edit and Submit a Job Scripts

● Go to the ~/exercises folder○ Check the README.en file for general instructions

● Start with the exercise 1-base. All the instructions are in the README.en file○ The solution is in solution.sh

● Then try exercises 2, 3 and/or 6○ For exercise 2, a description of “&” and “wait” is in

the README.en file of the exercise○ You need to understand the solution of the exercise

3 to do exercise 4

62

Compute Canada andCalcul Québec

63

Compute Canada

● Four consortia (from West to East):○ WestGrid

■ British Columbia, Alberta, Saskatchewan, Manitoba

○ Compute Ontario (CAC + SciNet + SHARCNET)■ Ontario

○ Calcul Québec■ Québec

○ ACENET■ New Brunswick, Newfoundland, Nova Scotia,

Prince Edward Island

64

Example - Calcul Québec Consortium

65

● Compute Canada○ Calcul Québec - Compute Resources

■ Briarée & Hadès (U. Montréal)■ Colosse & Helios (U. Laval)■ Guillimin (McGill)■ MS2 & MP2 (U. Sherbrooke)

○ Calcul Québec - Human Resources■ About 40 employees

● ARC analysts● System administrators● Scientific and operational directors

Biggest Clusters

66

Name / Institution Cores GPUs Summary

Niagara / U. Toronto ~60000 0 Specifications to be confirmed

Graham / U. Waterloo 33472 320 32 cores/node, 128+ GB/node, E/FDR IB

MP2 / U. Sherbrooke 30984 0 24 cores/node, 32+ GB/node, QDR IB

Cedar / SFU 27696 584 32 cores/node, 128+ GB/node, OPA 100

Guillimin / McGill U. 17240 112 12,16 cores/node, 24+ GB/node, QDR IB

Orcinus / U. British Columbia

9616 0 8,12 cores/node, 16+ GB/node, D/QDR IB

Orca / SHARCNET 8320 0 16,24 cores/node, 32 GB/node, QDR IB

Briarée / U. Montréal 7560 0 12 cores/node, 24+ GB/node, QDR IB

Example of job requirements: 8, 16 or 32 cores/node, 30 GB/node, QDR IB

Roles and Allocations

67

Prof

CCRI : abc-123-01 ABCD UniversityAlloc.: abc-123-aa (default alloc., Rapid Access Service or RAS)

CCRI : abc-123-02 DEFG UniversityAlloc.: abc-123-ac 200 core*years

Stu-dent

Prof

CCRI : def-456-01 HIJK UniversityAlloc.: def-456-ab 50 core*years

xyz-789-02

xyz-789-01

About Allocations

68

● Amount of compute cycles (only a target):○ 1 core provides 2 to 3 GHz (cycles per second)○ Core*seconds, core*hours or core*years (cy)○ Each CQ site has a default allocation (up to 80 cy)

● Amount of storage space (for CQ):○ Private scratch space (default: up to 1TB)○ Group shared project space (default: up to 1TB)

● Special allocations (RRG and RPP)○ Resource Allocation Competition (RAC 2018)https://www.computecanada.ca/research-portal/accessing-resources/resource-allocation-competitions/

How to Access CQ Resources - The Complete Procedure

● CCDB: https://ccdb.computecanada.ca● A professor creates an account in CCDB

○ Request role of Principal Investigator (PI) - CCRI○ (Optional) Request a special allocation (RAC)

■ Special allocation granted on a specific cluster● A student creates an account in CCDB

○ Request sponsored role - need to provide the PI’s CCRI

○ Apply for a consortium account (ex.: Calcul Québec)○ In the Calcul Québec portal, activate your access to

any cluster■ Choose clusters based on special allocations

69

Usage Policy

● Each user has his own account; never share your private account!

● No heavy process (in time and space) on login nodes

● Resources must be used in an optimal way○ Large parallel jobs must use tools that scale well○ “Cold” data should not stay on disk○ Maximize the amount of research done in Canada

● Must acknowledge Compute Canada and any regional partner organization

70

Service and Support● Training workshops:

○ Future: http://calculquebec.eventbrite.ca○ Past:

■ https://wiki.calculquebec.ca/w/Formations/en■ http://www.hpc.mcgill.ca/index.php/training

● Documentation:○ https://www.computecanada.ca/research-portal/○ https://docs.computecanada.ca/wiki/Compute_Canada_

Documentation○ https://wiki.calculquebec.ca/w/Accueil○ http://www.hpc.mcgill.ca/index.php/starthere

● Support:○ support@calculquebec.ca

71

Conclusion

72

Conclusion

● Need help?○ WestGrid: support@westgrid.ca○ SciNet: support@scinet.utoronto.ca○ SHARCNET: https://www.sharcnet.ca/my/security/login○ Calcul Québec: support@calculquebec.ca○ ACENET: support@ace-net.ca

● Useful information to send us:○ User name○ Job ID (if applicable)○ Path to {job script, output/error files, code} or as

attached files

73