Benchmarks of medical dosimetry simulation on the grid S. Chauvie 1, A. Lechner 4, P. Mendez Lorenzo...
-
Upload
bethany-gilmore -
Category
Documents
-
view
219 -
download
0
Transcript of Benchmarks of medical dosimetry simulation on the grid S. Chauvie 1, A. Lechner 4, P. Mendez Lorenzo...
Benchmarks of medical dosimetry simulation on the grid
S. Chauvie1, A. Lechner4, P. Mendez Lorenzo5, J. Moscicki5, M.G. Pia6
G.A.P. Cirrone2, G. Cuttone2, F. Di Rosa2, F. Foppiano3, M. Piergentili6, G. Russo2
Thanks for contributing to the development of Geant4 medical physics Advanced Examples!
1 Santa Croce and Carle Hospital, Cuneo, Italy
2INFN Laboratori Nazionali del Sud, Italy3IST, Genova, Italy
4Technical University Vienna, Austria 5CERN, Geneva, Switzerland
6INFN Genova, Italy
IEEE NSS 2007Honolulu, HI, USA
27 October – 3 November 2007
Monte Carlo methods in oncological radiotherapy
More precise than approximated analytical methods More precise than approximated analytical methods used in commercial treatment planning software systemsused in commercial treatment planning software systems
BUT
Too slow to be realistically usable in the clinical practiceToo slow to be realistically usable in the clinical practice
Variance reduction techniques Variance reduction techniques (event biasing)(event biasing)
Inverse Monte Carlo methodsInverse Monte Carlo methods
Analytical transport methodsAnalytical transport methods
“ “Fast simulation” techniquesFast simulation” techniques
ParallelisationParallelisation
Reduce simulation Reduce simulation execution timeexecution time
The GRID?
This projectThis project
Practical approach to the problem
Let’s take a few representative dosimetry simulation use casesLet’s run them on the gridLet’s evaluate (quantitatively) gains and drawbacks in realistic situations
Three Geant4 Advanced examples of dosimetry simulation covering major techniques in oncological radiotherapy
– Brachytherapy– Hadrontherapy– LINAC for IMRT (Intensity Modulated RadioTherapy)
LCG infrastructure, Geant4 Virtual Organization– The exercise will be repeated one step further as Mr. Nobody
No introduction on grid, EGEE and other flavours, LCG, medical-related grid projects, their impact etc.
brachytherapybrachytherapy
Leipzig applicator
MicroSelectron-HDR source
3 m m ste e l c a b le
5.0 m m
0.6 m m
3.5 m m
1.1 m m
Ac tive Ir-192 C o re
0 10 20 30 40 500,0
0,2
0,4
0,6
0,8
1,0
1,2 Simulazione Nucletron Misure
Dose %
Distanza lungo Z (mm)Distance along Z (mm)
SimulationNucletronDataExp. data: F. Foppiano et al.,
IST Genova
Relatively “fast” simulation
~ 7 CPU hours on an “average” PC to produce meaningful statistics for
clinical studies
Italian National Institute for
Cancer Research
Geant4 brachytherapy Advanced Example
hadrontherapyhadrontherapy
Accurate reproduction of the CATANA hadrontherapy
facility at INFN LNS, Catania, Sicily, Italy
Geant4 hadrontherapy Advanced Example
Geant4 geometry and beam primary generator implementation developed by INFN-LNS CATANA co-authors(Partial) code review by A.L. and MGP
See related talk in Event Processing
Session
The experimental data shown in this plot have been taken at the CATANA hadrontherapy
facility at INFN LNS, Catania by G.A.P. Cirrone, G. Cuttone, F. Di Rosa, G. Russo.
No credit to other co-authors and their institutes for these data.
Production aimed at validating physics modelling of the simulation
~ 150 CPU hours on an “average” PC to produce meaningful physics results
~ 16 hours for calibration results
Medical LINAC for IMRTMedical LINAC for IMRT
High demand of CPU resources for meaningful statistics
(e.g. for treatment planning verification)
≈ tens of CPU-days
Geometry developed by M. Piergentili
Italian National Institute for
Cancer Research
Exp. data: F. Foppiano et al., IST Genova
Geant4 medical_linac Advanced Example
RequirementsRequirements
Submit jobs to a Grid system
Plug-in Geant4 application
Split into tasks (without biasing the results)
Schedule tasks intelligently
Hide the underlying complexity to the end user
DIANE
Ganga
LCG
Geant4AIDAiAIDA
+ +
++
DIANEDIANEGeant4Geant4InterfacInterfacee
Interface class which binds together
Geant4 application and DIANE framework
DIANEMaster-Worker
architecture
Computing environmentComputing environmentIdentification of resources
– The site must support the Virtual Organization (Geant4)– Shared file system accessible through VO_GEANT4_SW_DIR
environment variable– Operating system and architecture compatible with Geant4
Software installation– Geant4, CLHEP, AIDA, gcc– Problems encountered: missing write permission, user authentication etc.– Requires familiarity with grid middleware
Requires experienced userOne must do one’s own installation on the grid
Interface to grid middleware (e.g. lxplus) must be set-up, otherwise additional (not trivial) installation needed (e.g. gLite)
Problem set-upProblem set-up
Determination of the total # of events to be produced– Defined for each application on the basis of the desired statistical
precision of dosimetric variables relevant to each use case
Splitting into tasks – Task = smaller group of events to be produced
Perform tasks sequentially on a worker node
Use a range of worker nodes in parallel in a grid environment
Critical issue
Task splittingTask splitting
Important for applications with short execution time – Time elapsed between submission of grid jobs and
beginning execution on a worker node comparable to total duration
High granularity → more effective use of CPU resources– Drawbacks: in general, larger output size to deal with
Task size(execution time)
# workers involved
balance
Splitting optimizationSplitting optimization
Goal: fastest simulation results
Many small tasks – 20000 events, ~ 25 s each
40 worker nodes
Higher # nodes would not be more effective
– Time for last registered workers to become active might be higher than total duration
Goal: sensitive simulation study– beam energy calibration
1 task = 1 worker– Not the fastest option
Compromise with large disk space requirements for output
Simplify distribution of random number seeds passed to tasks
“Fast” use casebrachytherapy
“Intensive” use casehadrontherapy
“One size DOES NOT fit all”
Baby-sittingBaby-sittingSelection of resources
– Availability, priority
Set-up a list of Computing Elements– Passed to Ganga through DIANE
Criteria for selection– State of Computing Element (production, draining mode)– Number of CPU allowed to VO at a given site– Estimated waiting time in batch queue– Estimated additional workload on given nodes– CPU speed
Relatively easy retrieval of information– Used for selection criteria
Must be done on a
regular basis
Automated tools not always effective
Effects of delayed worker registrationEffects of delayed worker registration
fast worker registration
aslow worker registration
b
Unsatisfactory conditions: delayed worker registration,
worker errors (crash, network problems etc.)
c
c
b
abrachytherapy
Effects more relevant to “fast” use cases
Effects of slow Effects of slow worker nodesworker nodes
More visible when few worker nodes
Similar execution time of all tasks on
all nodes
One slow node doubles the
total simulation duration
Both simulations executed in the same environment in
identicalComputing Elements
Possible explanation: variable workload on the
same nodes
““Fast” use case: brachytherapyFast” use case: brachytherapy
Sequential
1 task = 25 ± 0.5 CPU s
Total simulation = 417 ± 8 min on lcgui003
Task duration
Total simulation duration
~ factor 6 between min
and max duration
~ factor 4 between min
and max duration
40 DIANE workers
On the grid
64% runs terminated < 30 min96% runs terminated < 40 min
Ideal expectation with 40 workers: ~10 min for the whole simulation
More computationally intensive use More computationally intensive use case: hadrontherapycase: hadrontherapy
Sequential
1 task = 50’ ± 50 s CPU time
Total simulation = 16.7 h ± 17 min
20 DIANE workers
On the grid
84% runs terminated < 100 min
Ideal expectation with 20 workers: ~50 min for the whole simulation
Large scale medical Large scale medical simulationssimulations
Test case: hadrontherapy– Increase statistics by 2 orders of magnitude– 10M events for high precision physics studies– Expected duration of whole production: 2-3 days
3 large statistics runs– 2 did not complete (80% done)– 1 terminated successfully
Medical linac– Even more demanding than high statistics hadrontherapy– “Low” statistics demonstration OK– Realistic scale stable production does not look practical yet
ConclusionConclusion
Politically correct
Grid technology offers unprecedented opportunities to the medical physics community for fast,
high precision Monte Carlo simulation
Politically incorrect
Grid technology is not mature yet for stable operation in a clinical
environment by non-expert users
Scientific
Both statements above reflect reality
Quantitative tests evaluate the extent of success and contribute to identify
problems and bottlenecks
Measurements in real-life use cases are the path to solve problems
Anton Lechner
Austrian Doctoral StudentCERN IT/PSS
A BIG thank you!
Jürgen Knobloch Patricia Mendez Lorenzo Kuba Moscicki
Major contribution to the results presented!
and
Hurng-Chung Lee
Alberto Ribon
Alfonso ManteroGraduate Student at
INFN Genova1st Geant4-DIANE project
CERN IT/PSS
IEEE Transactions on Nuclear ScienceIEEE Transactions on Nuclear Sciencehttp://ieeexplore.ieee.org/xpl/RecentIssue.jsp?puNumber=23http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?puNumber=23
Prime journal on technology in particle/nuclear physics
Rigorous review process Associate Editor dedicated to computing papers
Various papers associated to CHEP 2006 published on IEEE TNS
Computing-related papers are welcomeComputing-related papers are welcome
Manuscript submission: http://tns-ieee.manuscriptcentral.com/
Publications on refereed journals are beneficial not only to authors, but to the whole community of computing-oriented physicists
Our “hardware colleagues” have better established publication habits…
Further info: [email protected]