Large-scale accelerator simulations: Synergia on the Grid

turn 1

turn 27turn 19

turn 16

C++

C++

Synergia

Field solver(FFT,

multigrid)

single particleoptics/utilities

wrapper/job control

glue

input &lattice(MAD)

analysistools

results

beamstudies

Python

Fortran 90

C++Octave,

C++

softw

are

simulations

data

Synergia

● Simulate multi-particle physics in accelerators

● Computationally intensive– 1-10's of millions of macro

particles– 10's of thousands (or more) of

PDE solves● Massively parallel

– Clusters and supercomputers● 64-node Linux cluster typical● 512 processors at NERSC

C++

C++

Synergia

Field solver(FFT, multigrid)

single particleoptics/utilities

wrapper/job control

glue

input &lattice(MAD)

analysistools

results

beamstudies

Python

Fortran 90

C++

Octave,

C++

Running Synergia

(1) Few-person collaboration

(2) Simulations require complex input parameters

(3) Output consists of many files

(4) Need to take advantage of computing resources wherever they are available

Grid computing is the answer for (4), but the increase in complexity arising from (2) and (3) has to be

mitigated.The tools provided by Synergia allow the

scientist to do science without getting bogged down by bookkeeping.

Computing on the Grid

● Scientist uses local resources for most tasks

● Remote systems used for computationally-intensive tasks only

● In our case, the computationally intensive tasks are running the simulations and some analysis

job exportjob creationjob DBanalysis tools

importresults

importresults

Job creation

● Python-based system– Python not required on target

site● Job contains

– Batch input● created from template

– Input files● user-defined

– Utilities● clean output, pack output

– Description● human and machine readable

Job directory

batch file

input files

utility scripts

description

Goal is reproducibility

Managing job options

● Python module for command-line options

● Groups of options can be composed– General Synergia options– Batch options– Application-specific

options– etc.

● Command-line is stored for cut-and-paste modification

● Automatic command-line help generation

● Automatic human-readable summary

● Options for created jobs can be added to database

Job database

Job information is automatically entered in spreadsheet.

Results

● The measure of a scientific computing project is the science it produces

● The Synergia infrastructure has allowed us to produce more science with less time wasted on tedious tasks– Better utilization of resources– Less time spent bookkeeping– Fewer redundant simulation runs

The measure of a scientific computing project is the science it produces

Fermilab Booster Accelerator