Introduction to the NERSC HPCF NERSC User Services

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

Introduction to the NERSC HPCFNERSC User Services

Hardware, Software, & Usage

Mass Storage

Access & Connectivity


June, 2000 Introduction to NERSC - User Services Group

2

Hardware, part 1• Cray Parallel Vector Processor (PVP) Systems

– 96 CPUs, Shared-memory parallelism (Cray tasking, OpenMP);

– J90SE clock is 100 MHz; peak performance is 200 Mflops/cpu (~125, actual)

– SV1 clock is 300 MHz; peak performance is 1200 Mflops/cpu (~300, actual)

– J90Se and SV1 are not binary compatible

• Cray T3E MPP System– mcurie

– 692 PEs: 644 application, 33 command, 15 OS; 256 MB/PE

– PE clock is 450 MHz; peak performance is 900 Mflops/PE (~100, actual)

Name Type # CPUs Memory Purpose

killeen J90SE 32 1 GW (8 GB) Interactive, weekdays;

Batch, up to 256 MW, 24x7, subject to

weekday suspension

seymour SV1 16 1 GW (8 GB) Interactive, weekdays;

Batch, up to 256 MW, 24x7, subject to

weekday suspension

bhaskara SV1 24 1 GW (8 GB) Batch, up to 512 MW

franklin SV1 24 1 GW (8 GB) Batch, up to 512 MW



3

Hardware, part 2• IBM SP MPP System

– gseaborg, Phase 1

– 304 nodes (608 CPUs): 256 (512) compute, 8 (16) login, 16 (32) GPFS, 8 (16) network, 16 (32) service); 1 GB/node

– Node clock is 200 MHz; peak performance is 800Mflops per CPU (~200, actual)

– Phase 2 will be bigger and faster

• Visualization Server– escher; SGI Onyx 2

– 8 CPUs, 5 GB RAM, 2 graphic pipes

– CPU clock is 195 MHz; 2 simultaneous video streams

• Math Server– newton; Sun UltraSPARC-II

– 1 CPU, 512 MB RAM

– CPU clock is 248 MHz



4

Hardware, part 3

• Parallel Distributed Systems Facility (PDSF)– High Energy Physics facility for detector simulation and data analysis

– Multiple clustered systems; Intel Linux PCs, Sun Solaris workstations

• Energy Sciences Network (ESNet)– Major component of the Internet; ATM Backbone

– Specializing in information retrieval, infrastructure, and group collaboration

• High Performance Storage System (HPSS)– Multiple libraries, hierarchical disk and tape archive systems

– High speed transfers to NERSC systems

– Accessible from outside NERSC

– Multiple user interface utilities

– Directories for individual users and project groups



5

PVP: File Systems, part 1

• $HOME

– “permanent” (but not archival)

– 5 GB quota, regular backups, file migration

– local to killeen, NFS-mounted on seymour and batch systems

– poor performance for batch jobs• /u/repo/u10101• /Un/u10101• /u/ccc/u10101• /U0/u10101



6

PVP: File Systems, part 2• $TMPDIR

• temporary (created/destroyed each session)

• no quota (but NQS limits 10 GB - 40 GB)

• no backups, no migration

• local to each machine– high-performance RAID arrays

– system manages this for you

• A.K.A. $BIG

• /tmp• location of $TMPDIR• 14-day lifetime

• A.K.A. /big• you manage this for yourself



7

PVP: Environment, part 1

• Unicos• Shells

– Supported• sh• csh• ksh (same as sh)

– Unsupported• tcsh (get it by “module load tcsh”)• bash (get it by “ module load tools”)



8

PVP: Environment, part 2• Modules

– Found on many Unix systems

– Sets all or any of environment variables, aliases, executable search paths, man search paths, header file include paths, library load paths

– Exercise care modifying startup files!

• Cray’s PrgEnv is modules-driven

• Provided startup files are critical!– Add to .ext files, don’t clobber originals

– Append to paths, don’t set them, and this only if necessary

– If you mess up, no compilers, etc.

• Useful commands• module list• module avail• module load modfile• module display modfile• module help modfile



9

PVP: Environment, part 3• Programming

– Fortran 90 - f90– C/C++ - cc, CC– Assembler - as– Use compiler (f90, cc, CC) for linking also

• f90 file naming conventions– filename.f - fixed form Fortran-77 code– filename.F - fixed form Fortran-77 code, run preprocessor first– filename.f90 - free form Fortran 90 code– filename.F90 - free form Fortran 90 code, run preprocessor first

• Multiprocessing (aka multitasking, multithreading…)– setenv NCPUS 4 (csh)– export NCPUS=4 (ksh)

– "a.out: Command not found.”– ./a.out … (Note: No parallelism specified with execution)



10

PVP: Environment, part 4a• Execution modes

– Interactive serial• 10 hours on killeen and seymour

• 80 MW max memory

– Interactive parallel• No guarantee of real-time concurrency

– Batch queues

* = killeen, seymour, franklin, bhaskara ** = franklin, bhaskara• To see them: qstat -b

• Queues shuffled at night, and sometimes during the day

• Subject to change

Queue Memory Limit(MW)

CPU TimeLimit (HRS)per Process

CPU TimeLimit (HRS)per Request

Disk SpaceLimit (GB)

debug_small * 80 0.5 0.5 40

st_80 * 80 120 121 40

st_256 * 256 120 121 40

st_512 ** 512 12 13 40

mt_512 ** 512 120 121 40



11

PVP: Environment, part 4b

• Batch– User creates shell script (e.g., “myscript”)– Submits to NQE with “cqsub myscript”

• Returns NQE task id (e.g., “t1234”)

– NQE selects machine and forwards to NQS• Job remains pending (“NPend”) until resources available

– NQS runs the job• Assigns NQS request id (e.g., “5678.bhaskara”)

• Run job in appropriate batch queue

– Job log returned upon completion



12


• Libraries– Mathematics

• nag, imsl, slatec, lsode, harwell, etc.

– Graphics• ncar, gnuplot, etc.

– I/O• HDF, netCDF, etc.

– Applications• Amber, Ansys, Basis, Gamess, Gaussian, Nastran, etc.



13


• Tools– ja - job accounting– hpm - Hardware Performance Monitor– prof - Execution time profiler & viewer– flowtrace/flowview - Execution time profiler & viewer– atexpert - Autotasking performance predictor– f90 - Compiler feedback– totalview - Debugger (visual and line-oriented)



14

T3E: File Systems, part 1

• $HOME

– “permanent” (but not archival)

– 2 GB quota, regular backups, file migration

– poor performance for batch jobs• /u/repo/u10101• /Un/u10101• /u/ccc/u10101• /U0/u10101



15

T3E: File Systems, part 2

• $TMPDIR• temporary (created/destroyed each session)

• 75 GB quota (but NQS limits 4 GB - 32 GB)

• no backups, no migration– high-performance RAID arrays

– system manages this for you

– Can be used for parallel files

• /tmp• location of $TMPDIR• 14-day lifetime

• A.K.A. /big• you manage this for yourself



16

T3E: Environment, part 1

• UNICOS/mk

• Shells: sh/ksh, csh, tcsh– Supported:

• Sh• Csh• ksh (same as sh)

– Unsupported:• tcsh (get it by “module load tcsh”)

• Bash (get it by “module load tools”)



17


• Modules - manages user environment– Paths, Environment variables, Aliases, same as on PVP systems

• Cray’s PrgEnv is modules-driven

• Provided startup files are critical!– Add to .ext files, don’t clobber originals– Append to paths, don’t set them, and this only if necessary

– If you mess up, no compilers, etc.• Useful commands

• module list• module avail• module load modfile• module display modfile• module help modfile



18

T3E: Environment, part 3a

• Programming– Fortran 90: f90

– C/C++: cc, CC– Assembler: cam cam

– Use compiler (f90, cc, CC) for linking also

• Same naming conventions as on PVP systems– PGHPF - Portland group HPF

– KCC: Kuck and Assoc. C++;• Get it via “module load KCC”

• Multiprocessing– Execution in Single-Program, Multiple-Data (SPMD) Mode

– In Fortran 90, C, C++, all processors execute same program



19

T3E: Environment, part 3b• Executables - Malleable or Fixed

– specified in compilation and/or execution– f90 -Xnpes ... (e.g., -X64) creates “fixed” executable

• Always runs on same number of (application) processors

• Type ./a.out to run

– f90 -Xm... or without -X option creates “malleable” executable• ./a.out will run on command PE

• mpprun -n npes ./a.out runs on npes APP PEs

– Executing code can ask for:• Process id (from zero up)

– MPI_COMM_RANK(...)

• Total number of PEs– MPI_COMM_SIZE(...)

• PE or Process/Task ID used to establish “master/slave” identities, controlling execution



20

T3E: Environment, part 4a• Execution modes

– Interactive serial• < 60 minutes on one command PE, 20 MW max memory

– Interactive parallel• < 30 minutes on < 64 processors, 29 MW memory per PE

– Batch queues

• To see them: qstat -b

• Queues shuffled in at night

• Subject to change

Pipe Queue Batch Queue MinPE

MaxPE

MPP Time Limit(seconds / hours)

Serial CPU TimeLimit

(seconds / hours)

serial serial_short 1 1 - 14400 / 4:00

debug_small 2 32 2000 / 0:33 3600 / 1:00debug

debug_medium 33 128 600 / 0:10 1800 / 0:30

pe16 2 16 15000 / 4:10 3600 / 1:00

pe32 17 32 15000 / 4:10 3600 / 1:00

pe64 33 64 15000 / 4:10 3600 / 1:00

pe128 65 128 15000 / 4:10 3600 / 1:00

pe256 129 256 15000 / 4:10 3600 / 1:00

production

pe512 257 512 15000 / 4:10 3600 / 1:00

long128 64 128 44200 / 12:17 3600 / 1:00long

long256 129 256 44200 / 12:17 3600 / 1:00



21

T3E: Environment, part 4b• (Old, obsolete) Example of T3E management and queue scheduling

Time\pe's 0 128 256 384 512 640

01:00 PE512

03:00 PE256 PE128 pe128 Int32

07:00 Long128 PE128 PE64 PE64 PE32 PE32 Int64

19:00 GC 256 GC128 GC128 PE64

23:00 PE64



22

T3E: Environment, part 5• Math & graphics libraries, and application codes are similar to

those on the PVP systems

• Libraries are needed for communication:– MPI (Message-Passing Interface)

– PVM (Parallel Virtual Machine)

– SHMEM (SHared MEMory; non-portable)

– BLACS (Basic Linear Algebra Communication Subprograms)

– ScaLAPACK (SCAlable [parts of] LAPACK)

– LIBSCI (including parallel FFTs), NAG, IMSL

• I/O libraries– Cray’s FFIO

– NetCDF (NETwork Common Data Format)

– HDF (Hierarchical Data Format)



23


• Tools– Apprentice - finds performance problems and inefficiencies

– PAT - Performance analysis tool

– TAU - ACTS tuning and analysis utility

– Vampir - commercial trace generation and viewing utility

– Totalview - multiprocessing-aware debugger

– F90 - compiler feedback



24

SP: File Systems, part 1

• AIX is a Virtual Memory operating system – Each node has its own disks, with OS image, swap and paging

spaces, and scratch partitions .

• Two types of user-accessible file systems: – Large, globally accessible parallel file system, called GPFS

– Smaller node-local partitions



25


• Environment variables identify directories – $HOME - your personal home directory

• Located in GPFS, so globally available to all jobs

• Home directories are not currently backed up!

• Quotas: 4 GB, and 5000 inodes

– $SCRATCH - one of your temporary spaces • Located in GPFS

• Very large - 3.5 TB

• Transient - purged after session or job termination

– $TMPDIR - another of your temporary spaces • Local to a node

• Small - only 1 GB

• Not particularly fast

• Transient - purged on termination of creating session or batch job



26

SP: File Systems, part 3• Directly-specified directory paths can also be used

– /scratch - temporary space• Located in GPFS

• Very large

• Not purged at job termination

• Subject to immediate purge

• Quotas: 100 GB and 6000 inodes

• Your $SCRATCH directory is set up in /scratch/tmpdirs/{nodename}/tmpdir.{number}

where {number} is system-generated

– /scratch/{username} - user-created temporary space • Located in GPFS

• Large, fast, encouraged usage

• Not purged at job termination

• Subject to purge after 7 days, or as needed

• Quotas: 100 GB and 6000 inodes



27


– /scr - temporary space • Local to a node

• Small - only 1 GB

• Your session-local $TMPDIR is set up in /scr/tmpdir.{number}where {number} is system-generated

• Not user-accessible, except for $TMPDIR

– /tmp - System-owned temporary space • Local to a node

• Very small - 65 MB

• Intended for use by utilities, such as vi for temporary files

• Dangerous - DO NOT USE!

• If filled up, it can cause the node to crash!



28

SP: Environment, part 1

• IBM's AIX - a true virtual memory kernel – Not a single system image, as on the T3E

– Local implementation of module system

– No modules load by default

– Default shell is csh– Shell startup files (e.g., .login, .cshrc, etc.) are links; DON’T delete them!

– Customize extension files (e.g., .cshrc.ext), not startup files



29


• SP Idniosyncracies– All nodes have unique identities; different logins may put

you on different nodes– Must change password, shell, etc. on gsadmin node

– No incoming FTP allowed

– xterms should not originate on the SP

– Different sessions may be connected to different nodes

– High speed I/O is done differently from the T3E

– Processors are faster, but communication is slower, than on the T3E

– PFTP is faster than native FTP

– SSH access methods differ, slightly



30

SP: Environment, part 3a• Programming in Fortran

– Fortran - Fortran 77, Fortran 90, and Fortran 95

– Multiple "versions" of the XLF compiler • xlf, xlf90 for ordinary serial code • xlf_r, xlf90_r for multithreaded code (shared memory parallelism) • mpxlf90, mpxlf90_r for MPI-based parallel code

– Currently, must specify separate temporary directory for Fortran-90 “modules”xlf90 -qmoddir=$TMPDIR -I$TMPDIR modulesource.F source.F

– IBM's HPF (xlhpf) is also available



31

SP: Environment, part 3b

• Programming in C and C++ – C & C++ languages supported by IBM

– Multiple "versions" of the XLC compiler • cc, xlc for ordinary serial C code • xlC for ordinary serial C++ code • cc_r, xlc_r for multithreaded C code (shared memory parallelism) • xlC_r for multithreaded C++ code (shared memory parallelism) • mpcc for MPI-based parallel C code • mpCC for MPI-based parallel C++ code

– Kuck & Assoc. KCC also available in its own module



32

SP: Environment, part 4a• Execution

– Many ways to run codes: • serial, parallel

• shared-memory parallel, message-based parallel, hybrid • interactive, batch

– Serial execution is easy:./a.out <input_file >output_file

– Parallel execution - SPMD Mode, as with T3E • Uses POE, a supra-OS resource manager

• Uses Loadleveler to schedule execution

• There is some overlap in options specifiable to POE and LoadLeveler

• You can use one or both processors on each node– environment variables and batch options control this



33

SP: Environment, part 4b– Shared memory parallel execution

• Within a node, only

• OpenMP, Posix Threads, IBM SMP directives

– Message-based parallel execution • Across nodes and within a node

• MPI , PVM, LAPI, SHMEM (planned)

– Hybrid parallel execution• Threading and message passing

• Most likely to succeed: OpenMP and MPI

– Currently, MPI understands inter- vs. intra-node communication, and sends intra-node messages efficiently



34

SP: Environment, part 4c– Interactive execution

• Interactive jobs run on login nodes or compute nodes

• currently, there are 8 login nodes

– Serial execution is easy:./a.out <input_file >output_file

– Parallel exeuction involves POE:poe ./a.out -procs 4 <input_file >output_file

• Interactive parallel jobs may be rejected due to resource scarcity; no queueing

• By default, parallel interactive jobs use both processors on each node

– Batch execution • Batch jobs run on the compute nodes

• By default, parallel batch jobs use both processors on each node;– you will be charged for both, even if you override this

• Use Loadleveler utilities set to submit, monitor, cancel, etc.

– requires a script, specifying resource usage details, execution parameters, etc.

– Several job classes, for charging, resource limits: premium, regular, low;

– two job types - serial and parallel



35

SP: Environment, part 4d• SP Batch Queues and resource Limits

• Limits:– 3 jobs running

– 10 jobs considered for scheduling (idle)

– 30 jobs submitted

Class Max Nodes Max Processors Max Time Priority

debug 16 32 30 minutes 20000

premium 256 512 4 hours 10000

regular 256 512 4 hours 5000

low 256 512 4 hours 1

interactive 8 16 20 minutes 15000



36


• Libraries and Other Software– Java, Assembler

– Aztec, PETSc, ScaLAPACK

– Emacs

– Gaussian 98, NWChem

– GNU Utilities

– HDF, netCDF

– IMSL, NAG, LAPACK

– MASS, ESSL, PESSL

– NCAR Graphics

– TCL/TK



37

SP: Environment, part 6• Tools

– VT - vsualization tool for trace visualization and performance monitoring

– Xprofiler - graphical code structure and execution time monitoring

– Totalview - multiprocessing-aware debugger

– Other Debugging Tools• Totalview - available in its own MODULE;

• adb - general purpose debugger

• dbx - symbolic debugger for C, C++, Pascal, and FORTRAN programs

• pdbx - based on dbx, with functionality for parallel programming

– TAU - ACTS tuning and analysis utility - planned!

– Vampir - commercial trace generation and viewing utility - future!

– KAP Suite - future?

– PAPI - future?



38

HPSS Mass Storage

• HPSS– Hierarchical, flexible, powerful, performance-oriented

– Multiple user interfaces allow easy, flexible storage management

– Two distinct physical library systems• May be logically merged in future software release

– Accessible from any system from inside or outside NERSC• hpss.nersc.gov, archive.nersc.gov (from outside NERSC)• hpss, archive (from inside NERSC)

– Accessible via several utilities• HSI, PFTP, FTP

• Can be accessed interactively or from batch jobs

• Compatible with system maintenance utilities (“sleepers”)



39

HPSS Mass Storage• HPSS

– Allocated and accounted, just like CPU resources• Storage Resource Units (SRU’s)

• Open ended - you get charged, but not cut off, if you exceed your allocation

• “Project” spaces available, for easy group collaboration

– Used for system backups and user archives• hpss used for both purposes• archive is for user use only

– Has modern access control• DCE allows automatic authentication

• Special DCE accounts needed

– Not uniformly accessible from all NERSC systems• Problems with PFTP on the SP system

– Modern secure access methods are problematic• ftp tunneling doesn’t work (yet…)



40

Accessing NERSC• NERSC recognizes two connection contexts:

– Interaction (working on a computer)

– File transfer

• Use of SSH is required for interaction (telnet, rlogin are prohibited)– SSH is (mostly) standardized and widely available

– Most Unix & Linux systems come with it

– Commercial (and some freeware) versions available for Windows, Macs,

– SSH allows telnet-like terminal sessions, but protects account name and password with encryption

• simple and transparent to set up and use

• Can look and act like rlogin

– SSH can forward xterm connections• sets up a special “DISPLAY” environment variable

• encrypts the entire session, in both directions



41

Accessing NERSC• SSH is encouraged for file transfers

– SSH contains “scp”, which acts like “rcp”– scp encrypts login info and all transferred data

– SSH also allows secure control connections through “tunneling” or “forwarding”

– Here’s how tunneling is done:• Set up a terminal connection to a remote host with port forwarding enabled

• This specifies a port on your workstation that ssh will forward to another host

• FTP to the forwarded port - looks like you are ftp’ing to your own workstation

• Control connection (login process) is forwarded encrypted

• Data connections proceed as any ftp transfer would, unencrypted

• Ongoing SSH issues being investigated by NERSC staff– Not all firewalls allow ftp tunneling, without “passive” mode

– HPSS won’t accept tunneled ftp connections

– Workstation platform affects tunneling method

– Methods differ slightly on the SP• New options, must use xterm forwarding, no ftp tunneling...

– Different platforms accept different ciphers



42

Information Sources - NERSC Web Pages



43

Information Sources - On-Line Lecture Materials

Introduction to the NERSC HPCF NERSC User Services

Documents

Transcript of Introduction to the NERSC HPCF NERSC User Services