Ian C. Smith 1
description
Transcript of Ian C. Smith 1
Ian C. Smith1
A portal-based system for quality assurance of radiotherapy treatment plans using Grid-enabled High Performance Computing clusters
CR Baker2, V Panettieri3, C Addison1, AE Nahum3
1 Computing Services Dept, University of Liverpool; 2 Directorate of Medical Imaging and Radiotherapy, University of Liverpool; 3 Physics Department, Clatterbridge Centre for Oncology
Outline
Introduction to radiotherapy treatment planning
University of Liverpool Grid Computing Server (GCS)
GCS tools
Command line job submission using the GCS
UL-GRID Portal
Results
Future directions
Rationale
Routine radiotherapy treatment planning is constrained by lack of sufficiently powerful computing resources
Monte Carlo (MC) based codes can provide accurate absorbed dose calculations but are computationally demanding (single simulation can take 3 weeks on a desktop machine)
Fortunately MC methods are inherently parallel – can run on HPC resources and (for some codes) HTC resources
So far looked at looked running simulations on local and centrally funded HPC clusters in a user-friendly manner
Starting to look at using Condor pools
Radiotherapy codes
Two MC codes have been investigated to date: MCNPX (beta v2.7a)
general purpose transport code, tracks nearly all particles at nearly all energies (https://mcnpx.lanl.gov/).
parallel (MPI-based) code, only runs on clusters self contained – no need for pre- and post- processing steps
PENELOPE general purpose MC code implemented as a set of FORTRAN routines coupled electron-photon transport from 50 eV to 1 GeV in arbitrary materials
and complex geometries[1]. serial implementation, will run on clusters and Condor pools needs pre- and post- processing to set up input files and combine partial
results
Starting to look at EGSnrc / BEAMnrc / DOSXYZnrc[1] Salvat F, Fernández-Varea JM, Sempau J. PENELOPE, a code system for Monte Carlo simulation of electron and photon transport. France: OECD NuclearEnergy Agency, Issy-les-Moulineaux; 2008. ISBN 9264023011. Available in pdfformat at: http://www.nea.fr.
Simulation of an electron treatment: from the treatment head to the patient
(taken from Cygler et al)
Courtesy of Prof. A. Nahum (CCO)
Grid Computing Server / UL-GRID Portal
Grid Computing Server / UL-GRID software stack
Grid Computing Server tools single sign on to resources via MyProxy, use ulg-get-proxy (proxies
automatically renewed)
job management is very similar to local batch systems such as SGE: ulg-qsub, ulg-qstat, ulg-qdel etc
support for submitting large numbers of jobs, file staging and pre- and post- processing
job submission process is the same for all compute clusters (local or external)
utility tools1 provide simple Grid based extensions to standard UNIX commands: ulg-cp, ulg-ls, ulg-rm, ulg-mkdir etc.
status commands available e.g. ulg-status, ulg-rqstat
1 based on GROWL scripts from STFC Daresbury
PENELOPE (serial code) workflows
Rereasdasdascreate random seeds for N input files using
clonEasy[1]
combine N individual phase-space files
compute individual phase-space file
create random seeds for N input files using clonEasy[1]
stage-in phase-space file (only if necessary)
compute partial treatment simulation results
combine partial treatment simulation results using clonEasy[1]
repeat for other patients
phase-space filecalculation
patient treatment simulation
PortalHPC cluster
[1] Badal A and Sempau J 2006 A package of Linux scripts for the parallelization of Monte Carlo simulations Comput.Phys. Commun. 175 440–50
GCS job description files for PENELOPE (1)## create phase space file (PSF)#job_type = remote_script
host = ulgbc2total_jobs = 16name = penelopeLPO
pre_processor = /opt1/ulgrid/apps/penelope/seed_input_filespre_processor_arguments = penmain_acc6_LPO35_.in 16
indexed_input_files = penmain_acc6_LPO35_.ininput_files = spectrum_pE_6_LPO35.geo, 6MW_2.mat
executable = /usr/local/bin/run_penmainarguments= penmain_acc6_LPO35_INDEX.in penmain_LPO35_INDEX.out
log = mylogfile
GCS job description files for PENELOPE (2)## perform patient simulation using previously calculated phase space file (PSF)#job_type = remote_scripthost = ulgbc2name = penelopetotal_jobs = 10
pre_processor=/opt1/ulgrid/apps/penelope/seed_input_filespre_processor_arguments=penmain.in 10
staged_input_files=PSF_test.psfinput_remote_stage_dir=staging
input_files = water_phantom.geo,water.matindexed_input_files = penmain.in
executable = /usr/local/bin/run_penmainarguments= penmainINDEX.in penmainINDEX.out ics
log = penelope.log
Condor job files for PENELOPE # job description filegrid_resource = gt2 ulgbc2.liv.ac.uk/jobmanager-condorg universe = gridexecutable = /usr/local/bin/run_penmain arguments = penmain_acc6_LPO35_$(PROCESS).in penmain_LPO35_$(PROCESS).out ics_test+ulg_job_name = penelopeLPOlog = logtransfer_input_files = spectrum_pE_6_LPO35.geo, 6MW_2.mat, penmain_acc6_LPO35_$(PROCESS).intransfer_files = alwaystransfer_executable = FALSE GlobusRSL = (count=1)(job_type=remote_script) \
(input_working_directory=/condor_data/smithic/penelope/big_test/create_psf) \ (job_name=penelopeLPO)notification = neverqueue 16
# DAG fileJOB pre_process dummy1.subJOB staging penelopeLPO35.subSCRIPT PRE pre_process /opt1/ulgrid/apps/penelope/seed_input_filesPARENT pre_process CHILD staging
GCS job submission and monitoringsmithic(ulgp4)create_psf$ ulg-qsub penelopeLPO35 Grid job submitted successfully, Job ID is 125042
smithic(ulgp4)create_psf$ ulg-qstat
Job ID Job Name Owner State Cores Host------ -------- ----- ----- ----- ----
125015.0 penelopeLPO smithic pr 1 ulgbc2.liv.ac.uk125034.0 penelope vpanetti r 1 ulgbc2.liv.ac.uk125035.0 penelope vpanetti w 1 ulgbc2.liv.ac.uk125038.0 penelope smithic si 1 ulgbc2.liv.ac.uk125042.0 penelopeLPO smithic qw 1 ulgbc2.liv.ac.uk125043.0 mcnpx3 colinb r 64 ulgbc4.liv.ac.uk125044.0 mcnpx3 colinb r 64 lancs2.nw-grid.ac.uk125044.0 gamess_test bonarlaw r 32 ngs.rl.ac.uk
Lung treatment simulated with PENELOPE and penVOX 07
7 fields
PSF calculation 1.5 days (14 cores) approximately 1.5 million particles
Patient calculation 1.5 days for all 7 fields (single core) Statistical uncertainty 1% (1 sigma)
2.5cm diameter beam, full energy (~60 MeV at patient, ~3.2 cm range in water)
500 million histories0.5x0.5x5 mm voxels50keV proton cut-off
<1% statistical uncertainty in absorbed dose in high dose region (1s)
Bragg peak
Half-modulation
Proton absorbed dose in water using MCNPX
Future Directions
Provide support for BEAM[1] and DOSxyz[3] (based on the EGSnrc MC code [2])
Utilise Liverpool Windows Condor Pool for running PENELOPE jobs
Compare with other implementations e.g. RT-Grid.
References:
[1] 23D. W. Rogers, B. Faddegon, G. X. Ding, C. M. Ma, J. Wei, and T. Mackie, “BEAM: A Monte Carlo code to simulate radiotherapy treatment units,” Med. Phys. 22, 503–524 _1995_.
[2] Kawrakow and D. W. O. Rogers. The EGSnrc Code System: Monte Carlo simulation of electron and photon transport. Technical Report PIRS-701 (4th printing), National Research Council of Canada, Ottawa, Canada, 2003.
[3] Walters B, Kawrakow I and Rogers D W O 2007 DOSXYZnrc Users Manual Report PIRS 794 (Ottawa: National Research Council of Canada)