K.HarrisonCERN, 6th March 2003

GANGA: GAUDI/ATHENA AND GRID ALLIANCE

- Aims and design - Progress with low-level software- Progress with Graphical User Interface- Conclusions and next steps

6th March 2003 2

Aims

- The Indian goddess Ganga descended to Earth to flow as a river (English: Ganges) that carried lost souls to salvation- Ganga software is being developed jointly by ATLAS and LHCb to provide an interface for running Gaudi/Athena applications on the Grid Deal with all phases of a job life cycle: configuration, submission monitoring, error recovery, output collection, bookkeeping Carry jobs to the Grid underworld, and hopefully bring them back- Idea is that Ganga will have functionality analogous to a mail system, with jobs having a role similar to mails Make configuring a Gaudi/Athena job and running it on the Grid as easy as sending a mail

6th March 2003 3

Ganga: schematic representation

Gaudi/Athenaapplication

GangaG

UI/C

LI

JobOptionsAlgorithms

Collective&

ResourceGrid

Services

HistogramsMonitoringResults

6th March 2003 4

Design considerations

- Ganga should not reproduce what already exists, but should make use of, and complement, work from other projects, including AtCom, AthASK, DIAL and Grappa in ATLAS Should also follow, and contribute to, developments in Physicist Interface (PI) project of LCG - The design should be modular, and the different modules should be accessed via a thin interface layer implemented using a scripting language, with Python the current choice- Ganga should provide a set of tools that can be accessed from the command line (may be used in scripts), together with a local GUI and/or a web-based GUI that simplifies the use of these tools- Ganga should allow access to local resources as well as to the Grid

6th March 2003 5

Tentative Ganga architecture

Server

EDG UI

PYTHON SW BUS

XML RPC server

XML RPC module

GANGA Core Module

OS Module

Gaudi/Athena

GaudiPython PythonROOT

PYTHON SW BUS

GU

I

DB

Remote user

(client)

LAN/WAN

GRID

LRMS

Local Job DB

ProductionDB

BookkeepingDB

JobConfiguration

DB

6th March 2003 6

Low-level software

- In Ganga, components used to define a job are mapped to Python classes- The low-level software needs to provide representations of the most basic components, and must allow flexibility in manipulating them- Users will have access to low-level data and methods at the command line, but in most cases shouldn’t need them Common tasks will be simplified through higher-level functionality, available both at the command line and through a GUI- Low-level software presented here works, but is preliminary Details of component representations are still being discussed by development team

6th March 2003 7

Job components- In the preliminary implementation, the component breakdown is as follows: Job: a WorkFlow and a set of methods for submitting a script to a particular batch system For now, consider LSF Grid will be modelled as just another batch system WorkFlow: a series of WorkSteps, defining the sum of the actions to be performed when the job is run WorkStep: a set of sub-components providing all information necessary to run one instance of an executable WorkStep sub-components: Command, InputFile, OutputFile, CMTPackage (produces a library), CMTApplication (has an executable associated with it) Other sub-components to follow

6th March 2003 8

Defining and running jobs

- For now, consider case where experiment’s standard software is pre-installed at the site where the job runs (as for EDG), and user software has been configured (i.e. cmt requirements and job options are defined) Ganga should help with configuration in the future, but user will always have to do the work of developing his/her own analysis code- User software does not have to be pre-installed on remote site- Command sequences may be entered at Ganga prompt, or may be used in a Python script that imports the Ganga classes: from GangaKernel.GangaImport import

6th March 2003 9

Atlas example (Atlfast)

atlasSetup = GangaCommand(“source /afs/cern.ch/user/h/harrison/public/atlasSetup.sh”)atlfast = GangaCMTApplication(“TestRelease”, “TestRelease-00-00-15”,”athena.exe”, “run/AtlasfastOptions.txt”)atlfastOutput = GangaOutputFile(“atlfast.ntup”)workStep1 = GangaWorkStep([atlasSetup,atlfast,atlfastOutput])workFlow = GangaWorkFlow([workStep1])lsfJob = GangaLSFJob(“atlfastTest”,workFlow)lsfJob.build()lsfJob.run() At this level, there is less built-in intelligence than in ASK, but a lot of flexibility

6th March 2003 10

LHCb example (DaVinci)lhcbSetup = GangaCommand(“source /afs/cern.ch/lhcb/scripts/lhcbenv.sh”)daVinciSetup = GangaCommand(“source /afs/cern.ch/lhcb/scripts/ProjectEnv.sh DaVinci v7r2”)daVinci = GangaCMTApplication(“Phys/DaVinci”, “v7r2”,”DaVinci.exe”, “options/DaVinci.opts”)daVinciOutput = GangaOutputFile(“dvhistos.hbook”)workStep1 = GangaWorkStep([lhcbSetup,daVinciSetup,daVinci, daVinciOutput])workFlow = GangaWorkFlow([workStep1])lsfJob = GangaLSFJob(“daVinciTest”,workFlow)lsfJob.build()lsfJob.run() Command sequences for Atlas and LHCb are very similar

6th March 2003 11

Graphical User Interface

- GUI development has been undertaken by A.Soroko- Implementation is based on wxPython extension module- Current version is mainly illustrative Right buttons are there; full functionality will come later

6th March 2003 12

GUI: basic design

Embedded Python

interpreter

6th March 2003 13

GUI: job creation

Create a new job

6th March 2003 14

GUI: specifying WorkFlow

Give name and version of

the application

6th March 2003 15

GUI: defining job options

Job (application)

options downloaded

from DB

6th March 2003 16

GUI: defining batch system

Define batch

system

6th March 2003 17

GUI: defining resource requirements

Define necessary resources

6th March 2003 18

GUI: job submission

Submit the job (after

configuration)

6th March 2003 19

GUI: notification of job submission

Job has been submitted

successfully

6th March 2003 20

Conclusions and next steps

- Low-level tools that allow (pre-configured) jobs to be handled in a similar manner for Atlas and LHCb have been developed- Implementation of GUI is at an advanced stage- Jobs have been submitted successfully through Ganga to Cern LSF Ready to start experimenting with job submission to Grid- C.L.Tan is working to understand how AtCom and Ganga can best complement one another- C.E.Tull has started investigating possibilities for exploiting Grid Monitoring Architecture in Ganga- Ganga abstract has been accepted for presentation at CHEP03