Tutorial CST simulations on HTCondor

Sebastien Joly

06/04/2020

You need to work in AFS !

On Linux, using the command ssh –Y username@lxplus.cern.ch

On Windows with PuTTY, hostname : username@lxplus.cern.ch, Port : 22

Create directory (command : mkdir HTCondor) in your AFS home folder named "HTCondor" for later use

Example paths : /afs/cern.ch/user/s/sjoly/HTCondor /!\ Max 10Go quota in …/user/…

/afs/cern.ch/work/s/sjoly/HTCondor /!\ Max 100Go quota in …/user/…

Then create 4 folders IN this directory named output, error, log, results :

mkdir output error log results

Launch HTCondor jobs from this directory

First connect to lxplus

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 2

To run a job on HTCondor you need :

• CST file (no parameters sweep)

• Executable file (.sh file)

• Submit file (.sub file)

…

• (This tutorial)

What you need for a HTCondor job

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 3

How to export CST results as text files

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 4

Open the Post-

processing top tab, then

the Result Templates

tab under Tools.

How to export CST results as text files

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 5

The template based post-

processing tab has the drop

down option General 1D.

Then the second drop down

list has ASCII Export.

Ticking Export All 1D

Results and adding a

postfix of .txt is how this is

done. This makes CST write

the results into txt files

under a

./simulation_name/Export

folder, that is easier to

access for transferring.

/!\ DO THIS STEP IF YOU WANT RESULTS TEXT FILES /!\

This file will list all the commands HTCondor will have to do, it's a shell script.

You can run it locally on a terminal but it'll take a very long time obviously.

To understand it let's look at an example.

Executable (.sh) file

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 6

#!/bin/bash

$ export EOS_MGM_URL=root://eosuser.cern.ch

$ export HOME='/afs/cern.ch/user/s/sjoly/HTCondor’

$ cernbox_folder='/eos/user/s/sjoly/HTCondor’

$ file_name=“example"

Example .sh file (Part 1/3)

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 7

Don’t forget these lines to

make the script work !

Change both HOME and

cernbox_folder by your AFS

and EOS paths.

file_name should be both the name

of the CST file and of its folder.

$ eos cp "$cernbox_folder "/"$file_name".cst "$HOME"/"$file_name".cst

$ chmod +x "$file_name".cst

$ /afs/cern.ch/project/parc/cst2018/cst_design_environment -t -tw "$HOME"/"$file_name".cst --numthreads 24

Example .sh file (Part 2/3)

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 8

Copy CST file from your EOS folder to your AFS folder.

Change permission to run CST file.

Open file with CST PARTICLE STUDIO wakefield solver (-t –tw argument), more

arguments on a later slide

/!\ -- numthreads MUST match number of requested CPUs, here 24.

#Optional: copy data files in text format in a separate directory for easy access

$ cd "$HOME"/results/

$ mkdir "$file_name"

$ cp "$HOME"/"$file_name"/Export/Particle* "$HOME"/results/"$file_name“

# Compress and copy output back to EOS

$ cd "$HOME"

$ tar -cvf "$file_name"_HTCondor.tar ./"$file_name" ./"$file_name".cst

$ eos cp ./"$file_name"_HTCondor.tar "$cernbox_folder "/"$file_name"_HTCondor.tar

exit 0

Example .sh file (Part 3/3)

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 9

Lines to copy all impedance

and wakes text files to a results

folder.

Lines to

compress and

move back CST

file and folder to

EOS folder.

-m Starts CST MICROWAVE STUDIO

-t Starts CST PARTICLE STUDIO

-e Starts the eigenmode solver; only valid for -m

-tw Starts the wakefield solver; only valid for -t

Example to launch CST PARTICLE STUDIO wakefield solver :

/afs/cern.ch/project/parc/cst2018/cst_design_environment -t -tw file_name.cst

More arguments by typing this line on lxplus console :

/afs/cern.ch/project/parc/cst2018/cst_design_environment --help

CST arguments

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 10

This file allows you to run the commands in your .sh file on HTCondor clusters.

To submit a job you need to use the command :

condor_submit example.sub

To understand it let's look at an example.

Submit (.sub) file

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 11

executable = example.sh

RequestCpus = 24

+WCKey = CST

+JobFlavour = "workday"

environment = CST_INSTALLPATH="/afs/cern.ch/project/parc/cst2018"; CST_WAIT_FOR_LICENSE=on; CST_LICENSE_SERVER="1705@lxlicen01","1705@lxlicen02","1705@lxlicen03"; HOME="/afs/cern.ch/user/s/sjoly/HTCondor"

transfer_output_files = ""

output = output/output.$(ProcId).out

error = error/error.$(ProcId).err

log = log/log.$(ProcId).log

queue

Example .sub file (Complete)

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 12

Number of requested CPUs, should match -- numthreads in .sh file.

Job flavour corresponds to maximum run time of

the job, more info on a later slide.

Don’t forget to change AFS path here.

A CPU core corresponds to 2gb of memory and 20gb of disk space.

By default, a job will get one slot of a CPU core.

To ask for more CPUs you can do the following in the submit file:

RequestCpus = 4

This will result in a slot of 4 CPUs, 8gb of memory and 20gb of disk.

It's possible to ask for more memory but I haven't tried it.

RequestMemory = 14000

CPUs and memory

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 13

espresso = 20 minutes

microcentury = 1 hour

longlunch = 2 hours

workday = 8 hours

tomorrow = 1 day

testmatch = 3 days

nextweek = 1 week

Setting manually the maximum runtime can be achieved by placing the following in your submit file:

+MaxRuntime = Number of seconds

(Allow for runtime > 1 week)

Job flavours

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 14

• The "output" folder contains all the messages that would have otherwise been printed on CST in the "Progress" box. Very useful to see if a problem was caused by CST.

• The "error" folder contains any errors that would normally have been printed to the terminal if you were running the script/program yourself from the command line. Good place to look to see if the .sh file triggered an error.

• The "log" folder contains information that HTCondor tracks for each job, including when it was submitted, started, and stopped. It also describes resource use, and where the job ran. Can be interesting but less clear to read.

Output folders

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 15

• Change permissions on both your executable and submit file by doing :

chmod +x example.sh

chmod +x example.sub

• Try to run your .sh file on lxplus before submitting it to see if it works :

./example.sh

You can stop it once it’s starting the CST step

• You can follow CST progress by doing :

nano ./file_name/Result/progress.log

Good practices

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 16

Submit job : condor_submit example.sub

Check the status of the job execution: condor_q

Cancel a job: condor_rm JOBID

Access the job's local files while it is running: condor_ssh_to_job <jobid>

Check all the details about the job: condor_q -better {job-id}

Check how the job was executed once it finishes: condor_q JOBID -l

Check how long the job was running: condor_history JOBID -limit 1 -af:hRemoteWallClockTime

Useful HTCondor commands

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 17

EOS tutorial

How to run CST simulations in the Linux Cluster

HTCondor troubleshooting

References

06/04/2020 Sebastien Joly | Tutorial CST simulations on HTCondor 18

home.cern