Introduction to the T3E
description
Transcript of Introduction to the T3E
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 1
Introduction to the T3E
Mark Durst
NERSC/USG
ERSUG Training, Argonne, IL
28 April 1999
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 2
Outline
• Hardware and Configuration
• Programming Environment
• Planning Runs
• Monitoring Execution
• Accounting
• Additional Resources
• Elvis Impression
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 3
NERSC T3E Configuration
• Commodity DEC Alpha EV-5 superscalar processor– 450 MHz clock
– 900 Mflops/PE peak (only 5-10% typically achieved)• Theoretical peak performance: 575 Gflops
– 256 MB memory per PE
• 692 PEs in 3 flavors– 644 Application
– 33 Command (ideally)
– 15 OS
• Access via telnet, ssh, FTP
• Connect to NERSC mass storage, AFS
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 4
Interactive Environment
• UNICOS/mk
• Available shells: sh/ksh, csh, tcsh– csh: no file completion
– tcsh not Cray-supported
• Home directories– 2 GB file quota (with possible data migration)
– 3500 inode quota
• /usr/tmp – Used both for batch and temporary user space
– 75 GByte quota, 6K inode quota
– Fastest transfer rates
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 5
modules
• modules manages user environment– Paths
– Environment variables
– Aliases
• Cray’s PrgEnv is modules-driven
• Provided startup files are critical!– Add to them, don’t clobber them
– Add to paths, don’t set them
– If you mess up, no compilers, etc.
• Largely automatic
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 6
More Fun with modules
• module list (tells you what’s loaded)
• module avail (lists them all)
• Other module subcommands– load– unload– switch– help
• Roll back compilers
• Test new versions• http://home.nersc.gov/software/os/modules.html
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 7
Other modules
• imsl (loads by default) • nag (loads by default) • scalapack (1.5)
• GNU (prepends) and GNU.tools (appends)
• tools (tcsh, bash)• netcdf
• KCC (KAI C++ compiler)• USG• tedi
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 8
Programming Environment
• f90• cc/CC
• cam (assembler)
• cld (loader; usually unneeded)• pghpf
• KCC (“module load KCC”)
• totalview (debugger)
• pat, apprentice (performance analysis)
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 9
f90
• Conforms to Fortran 90 standard
• Much “standard” f77 wasn’t
• User-defined and abstract types
• Array syntax
• Allocatable objects and pointers
• Additional intrinsics
• cpp-like preprocessor
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 10
Important f90 options
• -f: source form (fixed or free)• Defaults: .f fixed, .f90 free
• -c: Compile only
• -o name: Name executable• Overrides -c (use -b name instead)
• -g, -G0, -G1: debugging
• -O[0-3]: general optimization
• -Ra, -Rb: Argument/Bounds checking
• -dp: Double precision 64-bit single precision
• -i 32 / -s default32: 32-bit integers / numbers
• -ev: Static memory allocation
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 11
Executables: Malleable or Fixed
• -Xnpes (e.g., -X64) creates “fixed” executable– Always runs on same number of (application) processors
– Type ./a.out to run
• -Xm or no -X option creates “malleable” executable– ./a.out will run on command PE
– mpprun -n npes ./a.out runs on npes APP PEs
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 12
Execution Model
• In F90, C, C++, all processors execute same program
• Can ask for:– Process number (from zero up)
• MY_PE() (F90)
• _my_pe() (C/C++)
– Total number of PEs• NUM_PES() (F90)
• _num_pes() (C/C++)
• Above used to establish “master/slave” relationships
• Libraries still needed for communication
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 13
Libraries
• MPI (Message-Passing Interface)
• PVM (Parallel Virtual Machine)
• SHMEM (SHared MEMory; non-portable)
• BLACS (Basic Linear Algebra Communication Subprograms)
• ScaLAPACK (SCAlable [parts of] LAPACK)
• NetCDF (NETwork Common Data Format)
• HDF (Hierarchical Data Format)
• LIBSCI (including parallel FFTs), NAG, IMSL
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 14
Archival Storage in HPSS
• High-Performance Storage System
• Designed for scalability & hierarchies
• User storage quotas exist
• Access via ftp or new hsi utility
• Two systems:– hpss.nersc.gov (hsi hpss)– archive.nersc.gov (hsi, hsi archive) contains old CFS
files
– merger planned
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 15
Networking Issues
• AFS– Accounts must be requested
– Tiny local quotas
– Available on Crays through NFS/AFS gateway• Non-trivial latencies
• Remote logins– .rhosts access not permitted; no incoming “r- commands”– ssh available
• xterm only “backwards”
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 16
Execution modes
• Interactive serial– < 60 minutes
– on command PEs
– slightly reduced memory
• Interactive parallel– < 30 minutes
– < 64 processors
• Batch
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 17
Batch queues on mcurie.nersc.gov
• To see them: qstat -b• pe16 through pe512
– 4 hours “on the torus”
– Routine parallel jobs
• serial_short: 4 hours on a single command PE
• debug_small: ½ hour, up to 32 PEs
• long128, gc128, gc256: 12-hour queues 64 PEs
– gc queues restricted
• Largest queues shuffled in at night• Other jobs checkpointed out
• Subject to change
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 18
Example daily job mix
Time\pe's 0 128 256 384 512 640
01:00 PE512
03:00 PE256 PE128 pe128 Int32
07:00 Long128 PE128 PE64 PE64 PE32 PE32 Int64
19:00 GC 256 GC128 GC128 PE64
23:00 PE64
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 19
Batch submission
• Jobs are shell scripts
• cqsub submits, returns task ID; cqdel deletes
• cqstatl/qstat gets status (many options)
• NQS parameters determine queue– #QSUB -l mpp_p=… (number of PEs)– #QSUB -l mpp_t=… (“parallel” time)
– for serial jobs:• use #QSUB -q serial• not #QSUB -l mpp_p=1
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 20
Pipe Queues
• You submit to pipe queues, not batch queues– Use only pipe names in directives like:
#QSUB -q serial
• Group batch queues:– serial = serial_short– debug = debug_small– production = pe128 through pe512– long = long128, gc128, gc256
• 3 jobs per user in production + long• 3 in serial, one in debug• To see them: qstat -p
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 21
Scheduling Information
• Lots of NQS-related limits– Queue run limits
– Queue “complex” run limits
• Global Resource Manager– Fits jobs into contiguous sets of PEs
– Once started, jobs run to completion (mostly)
– First-fit algorithm lets small jobs trample big ones
– grmview shows PE status, waiting jobs
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 22
Scheduling Information (cont’d)
• pslist gives summary of GRM data– No man page; pslist -h instead
• Checkpointing– For system maintenance
– To run test and “grand challenge” jobs
– Shows “Hop” in qstat/cqstatl (held by operator)
• mppview more nuts-and-bolts
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 23
Accounting and allocations
• T3E allocations are in node-minutes– setcub view repo=reponame– setcub view user=username
• newacct reponame switches repos interactively– One login name per user; multiple repos
• #QSUB -A reponame charges batch jobs • Charging updated daily; enforcement manual
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 24
On-line Resources
• T3E pages under “Computers” at home.nersc.gov– Read overview once, check “Changes” monthly
• Docs in Cray on-line system– http://www.cray.com/swpubs/
• “Topics” to T3E collection
– Many other docs (e.g., F90, C manual sets)
• Cray Web site, www.cray.com– Technical documents, additional on-line docs
• NERSC T3E tutorials– “Training” “NERSC Tutorials”
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 25
More on-line resources
• Other NERSC tutorials– Using the Cray f90 compiler at NERSC– Introduction to make
– NQE: Using the batch system
• Look over NERSC Web generally
NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER
INTRODUCTION TO THE T3E SYSTEM 26
man pages
• cqsub• cqstatl• f90 • cc• CC