Advanced Parallel Computing for Scientific ApplicationsAutumn Term 2010

Prof. I. F. SbalzariniETH Zentrum, CAB G34CH-8092 Zurich

Getting started on Brutus

This little reference is just meant to help you getting started on Brutus. It only covers the very basics thatare necessary to start working on the exercises. I am sure there will be more questions coming up once youare working. The first address when looking for answers is the Brutus Wiki, it’s easy to understand andcontinuously updated:

http://brutuswiki.ethz.ch

(then follow the link to Brutus)

Login to Brutus:Please visit brutuswiki.ethz.ch/brutus/Accessing the cluster.

Open a terminal (Linux or UNIX) and enter the following command:ssh <yourusername>@brutus.ethz.chAfter entering your password, you should now be in your home directory via one of the Brutus loginnodes. By using the option -Y when logging in, you set up an X11 session such that opening windowsare displayed on your screen.

If you are working under Microsoft Windows, you have to install an SSH client, e.g. PuTTY, whereyou can start an interactive login session. Use brutus.ethz.ch as the Host Name and choose SSH

as the Connection Type. As an alternative to using PuTTY, you can install Cygwin, a Linux-likeenvironment for Windows.

Copying a file to/from Brutus:Please visit brutuswiki.ethz.ch/brutus/File transfer.

When using a Linux or Unix terminal (also via Mac):Type in your local terminal:scp <pathorig>/<origfile> <username>@brutus.ethz.ch:<pathdest>/<destfile> orscp <username>@brutus.ethz.ch:<pathorig>/<origfile> <pathdest>/<destfile> to copyto or from Brutus from or to your local workstation.Or, type in the Brutus terminal:scp <username>@<localhost>:<pathorig>/<origfile> <pathdest>/<destfile> orscp <pathorig>/<origfile> <username>@<localhost>:<pathdest>/<destfile> to copy toor from Brutus from or to your local workstation.The option -c blowfish will speed up file transfer.

When working with Microsoft Windows:Clients like FileZilla (also available for Linux or Mac) support secure file transport using a GUI.

On Macs only:The application Macfusion allows you to mount your Brutus home in your Finder such that you canopen/modify files on Brutus with your local applications. Requires OS X 10.5 Leopard.

1

Editing a text file on Brutus:

You can choose between the text editors vi, nedit, and emacs. NEdit and Emacs will have a GUI ifyou established an X11 session. NEdit does not work without GUI.

Compiling and running your program on Brutus:Please visit brutuswiki.ethz.ch/brutus/Compiling an application andbrutuswiki.ethz.ch/brutus/Using the batch system

Serial code (no usage of MPI):Go to the directory your source code lies in. Compile with gcc <srcfile> -o <myprog>. Submitthe job to the queuing system with bsub ./<myprog>. You can check the status (pending or running)of your job with the command bjobs.

Please, DO NOT RUN YOUR APPLICATION ON THE LOGIN NODE! Login nodesare only the gateway to the cluster. You are allowed to run short programs (CPU time < 1 min)interactively (use the -I option) on the login nodes for testing and debugging purposes, or for quickpre- or post-processing. Anything else is prohibited.

Parallel code using OpenMP:Todays C and Fortran compiler support OpenMP. To compile an OpenMP source file you can addthe -fopenmp flag to the compiler. Before submitting, type export OMP NUM THREADS=<N> where Nis the number of threads you like to use. To submit the parallel job to Brutus’ batch systems, thecommand looks like follows for N threads: bsub -n N ./<my openmp prog>

Parallel code using MPI:Before you compile your code, you have to load the MPI module:module load open mpi

(Modules affect only the shell in which the module command is executed and are unloaded when youlogout. To see which modules you have loaded, use the command module list.) Now you compileyour code like the serial code, you just have to replace the command gcc by mpicc. Submitting thejob is a bit different because you have to specify that you want to run a parallel program and howmany processes you want. The command is bsub -n N mpirun ./<myprog> where N is the numberof processes. When you submit a job, the batch system analyzes its requirements (run-time, numberof processors, memory, communication network) and dispatches it to the right queue. This processis fully automatic. By default there is a run-time limit of one hour. If you want to execute a longerjob, use the option -W HH:MM. Also, use the -W flag if you have a short job that you like to test onBrutus since short jobs are scheduled more often.

Submit a shell script:Please submit your shellscript as follows: bsub < script. Don’t use: bsub /path/to/script

Checking on your jobs:

To see all of your current jobs, their status (pending or running), their job-ID:bjobs or bjobs -l for additional details.Impressive: the option -u all shows you all users’ jobs in the batch system...

To kill one of your jobs: bkill <jobID>

To kill all your jobs: bkill 0

Once a job is finished you will find a file called lsf.o<jobID> in your working directory. It containsinformation about the completed job (when, from where submitted, how long did it run etc.), andthe program’s output and error messages that have been sent to standard output.

2

Logout from Brutus:

When you want to stop your connection to Brutus, use the command logout or exit. The logoutdoes not work if you have still applications running on the login node, however, your jobs waitingin the queuing system or running on the compute nodes are not affected by and do not inhibit yourlogout.

3