Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation:...
-
Upload
xavier-weeks -
Category
Documents
-
view
220 -
download
0
Transcript of Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation:...
![Page 1: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/1.jpg)
Maria Grazia Pia, INFN Genova
Epistemic and systematic uncertainties in Monte Carlo simulation:
an investigation in proton Bragg peak simulation
Maria Grazia Pia INFN Genova, Italy
Maria Grazia Pia1, Marcia Begalli2, Anton Lechner3, Lina Quintieri4, Paolo Saracco1
1 INFN Sezione di Genova, Italy2 State University Rio de Janeiro, Brazil
3 Vienna University of Technology, Austria4 INFN Laboratori Nazionali di Frascati,, Italy
SNA + MC 2010Joint International Conference on
Supercomputing in Nuclear Applications + Monte Carlo 2010
![Page 2: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/2.jpg)
Maria Grazia Pia, INFN Genova
Quantifying the unknown in Monte Carlo simulation
Maria Grazia Pia INFN Genova, Italy
Maria Grazia Pia1, Marcia Begalli2, Anton Lechner3, Lina Quintieri4, Paolo Saracco1
1 INFN Sezione di Genova, Italy2 State University Rio de Janeiro, Brazil
3 Vienna University of Technology, Austria4 INFN Laboratori Nazionali di Frascati,, Italy
SNA + MC 2010Joint International Conference on
Supercomputing in Nuclear Applications + Monte Carlo 2010
![Page 3: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/3.jpg)
Maria Grazia Pia, INFN Genova
Epistemic uncertainties
Possible sources in Monte Carlo simulationincomplete understanding of fundamental physics processes, or practical inability to treat them thoroughly
non-existent or conflicting experimental data for a physical parameter or model
applying a physics model beyond the experimental conditions in which its validity has been demonstrated
Epistemic uncertainties originate from lack of knowledge
Epistemic uncertainties affect the reliability of simulation results
Can we quantify them?
Relatively scarce attention so far in Monte Carlo simulationStudies in deterministic simulation (especially for critical applications)
![Page 4: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/4.jpg)
Maria Grazia Pia, INFN Genova
Uncertainty quantification
Epistemic uncertainties are difficult to quantify due to their intrinsic nature
No generally accepted method of measuring epistemic uncertainties and their contributions to reliability estimation
Various formalisms developed in the field of deterministic simulation Interval analysis Dempster-Shafer theory of evidence
Not always directly applicable in Monte Carlo simulation Adapt, reinterpret, reformulate existing formalisms Develop new ones specific to Monte Carlo simulation
![Page 5: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/5.jpg)
Maria Grazia Pia, INFN Genova
Benefits of quantifying uncertainties
Epistemic uncertainties are reducible Can be reduced or suppressed by extending knowledge New experimental measurements
Uncertainty quantification gives us guidance about What to measure What experimental precision is needed/adequate Priorities: which uncertainties generate the worst systematic
effects
Measurements are not always practically possible Uncertainty quantification to control systematics
![Page 6: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/6.jpg)
Maria Grazia Pia, INFN Genova
Warm-up exercise
Epistemic uncertainties quantification in proton depth dose simulation
simplicity complexity
![Page 7: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/7.jpg)
Maria Grazia Pia, INFN Genova
Ingredients
p stopping powers
Water ionisation potential
d-ray production
Multiple scattering
Nuclear elastic
Nuclear inelastic Cross sections Preequilibrium Deexcitation Intranuclear cascade
EGS5, EGSnrc
Penelope
MCNP(X)
PHITS
SHIELD-HIT
FLUKA
GEANT 3
SPAR, CALOR, CEM, LAHET, INUCL, GHEISHA, Liège INCL, Bertini
d-ray or no d-ray
Preequilibrium or no preequilibrium
Weisskopf-Ewing or Weisskopf-Ewing
Griffin-exciton or hybrid
etc.
![Page 8: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/8.jpg)
Maria Grazia Pia, INFN Genova
Geant4 physics options
Water ionization potential set through the public interface of G4Material
![Page 9: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/9.jpg)
Maria Grazia Pia, INFN Genova
“Validation” in the literature
Beam energy (and energy spread) is not usually known with adequate precision in therapeutical beam lines What matters in clinical applications is the range
Typical procedure: optimize the beam parameters to be used in the simulation by fitting them to experimental data Determine beam energy, energy spread etc. Use optimized beam parameter values in the simulation
This is a calibration
This is NOT validation
T. G. Trucano, L. P. Swiler, T. Igusa, W. L. Oberkampf, and M. Pilch,“Calibration, validation, and sensitivity analysis: What’s what”,
Reliab. Eng. Syst. Safety, vol. 91, no. 10-11, pp. 1331-1357, 2006.
![Page 10: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/10.jpg)
Maria Grazia Pia, INFN Genova
Simulation environment
Realistic proton beam line Geometry from Geant4 hadrontherapy advanced example G. A. P. Cirrone, G. Cuttone, S. Guatelli, S. Lo Nigro, B. Mascialino, M. G. Pia, L. Raffaele,
G. Russo, M. G. Sabini,“Implementation of a New Monte Carlo GEANT4 Simulation Tool for the Development of a Proton Therapy Beam Line and Verification of the Related Dose Distributions”, IEEE Trans. Nucl. Sci., vol. 52, no. 1, pp. 262-265, 2005
Water sensitive volume, longitudinal 200 mm slices (through G4ReadoutGeometry)
Proton beam: E = 63.95 MeV, sE = 300 keV
Physics modeling options configured through an application design based on G4VModularPhysicsList
1 million primary protons
Geant4 8.1p02, 9.1(ref-04), 9.2p03, 9.3
![Page 11: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/11.jpg)
Maria Grazia Pia, INFN Genova
Reference physics configuration
Wellisch & Axen
Wellisch & Axen
![Page 12: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/12.jpg)
Maria Grazia Pia, INFN Genova
General featureselectromagneticelectromagnetic + hadronic elasticelectromagnetic + hadronic elastic + hadronic inelastic
electrons
59.823 MeV peaks=376 keV
![Page 13: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/13.jpg)
Maria Grazia Pia, INFN Genova
Water mean ionisation potential
Ep = 63.95 MeVI = 75 eV, 67.2 eV, 80.8 eV
Ep = 63.65 MeV (1s from 63.95 MeV)
I = 80.8 eV
GoF tests Bragg-Braggp-value = 1 (Kolmogorov-Smirnov, Anderson-Darling, Cramer-von Mises)
![Page 14: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/14.jpg)
Maria Grazia Pia, INFN Genova
Proton stopping powers
ICRU49Ziegler77Ziegler85Ziegler2000
Differences would be masked by typical calibration of simulation input parameters
![Page 15: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/15.jpg)
Maria Grazia Pia, INFN Genova
Hadronic elastic scattering
U-elastic Bertini-elasticLEP (GHEISHA-like)CHIPS-elastic
p-value (reference: U-elastic)
Bertini LEP CHIPS
Wald-Wolfowitz test: p-value< 0.001
Difference of deposited energy in
longitudinal slices
![Page 16: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/16.jpg)
Maria Grazia Pia, INFN Genova
Hadronic inelastic cross sections
GHEISHA-like Wellisch & Axen
Difference of deposited energy
in longitudinal slices
99% confidence interval for inelastic scattering occurrences in water (Wellisch & Axen cross sections): 1688-1849
Occurrences with GHEISHA-like cross sections: 1654
Bragg peak profilesp-value > 0.9
(Kolmogorov-Smirnov, Anderson-Darling, Cramer-von Mises)
![Page 17: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/17.jpg)
Maria Grazia Pia, INFN Genova
Hadronic inelastic scattering models
No visible difference in Bragg peak profiles
Wald-Wolfowitz test
p-value< 0.001
for all model options except
p-value=0.360for Liège cascade
p-value (reference: Precompound)
preequilibrium = no preequilibrium
![Page 18: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/18.jpg)
Maria Grazia Pia, INFN Genova
Hadronic inelastic differences
Difference of deposited energy in
longitudinal slices
Bertini LEP Liège CHIPS
reference: Precompound
secondary p
secondary n
Precompound Bertini
LEPLiège
CHIPS
Precompound Bertini
LEPLiège
CHIPSWald-Wolfowitz test: p-value < 0.001
![Page 19: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/19.jpg)
Maria Grazia Pia, INFN Genova
Nuclear deexcitation
reference: default Evaporation
GEM evaporation
Fermi break-up
Difference of deposited energy in
longitudinal slices
Difference of deposited energy in
longitudinal slices
Geant4 < 9.3(bug fix)
default evaporation GEM evaporation
Fermi break up Binary Cascade
![Page 20: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/20.jpg)
Maria Grazia Pia, INFN Genova
Cascade-preequilibrium
Precompound model activated through Binary Cascade w.r.t. standalone Precompound model
Difference of deposited energy in
longitudinal slices
systematic effect
In Geant4 Binary Cascade model cascading continues
as long as there are particles above a 70 MeV kinetic energy threshold
(along with other conditions required by the algorithm)
Transition between intranuclear cascade and
preequilibrium determined by empirical considerations
![Page 21: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/21.jpg)
Maria Grazia Pia, INFN Genova
8.1 9.1 9.2.p03 9.3 9.3 hMS
3.9
4.0
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
Geant4 version
Acc
epta
nce
(%
)
Some get lost on the way…
95% confidence
intervals
July 2006
December 2009
Calibration: 50 and 200 GeV
![Page 22: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/22.jpg)
Maria Grazia Pia, INFN Genova
Multiple scatteringRangeFactor StepLimit LatDisplacement skin geomFactor Model
8.1 0.02 1 UrbanMSC9.1 0.02 1 1 0 2.5 UrbanMSC
9.2p0.3 0.02 1 1 3 2.5 UrbanMSC9.3 0.04 1 1 3 2.5 UrbanMsc92
9.3 hMS 0.2 0 1 3 2.5 UrbanMsc90
G4MultipleScatteringG4hMultipleScattering
G4hMultipleScattering, Geant4 9.3 G4MultipleScattering, Geant4 9.3G4MultipleScattering, Geant4 9.2p03 G4MultipleScattering, Geant4 9.1 G4MultipleScattering, Geant4 8.1p02
Reference: Geant4 9.3 G4hMultipleScattering
Difference: G4MultipleScattering in Geant4 9.3 9.1 9.2p03 8.1p02
Difference of deposited energy in
longitudinal slices
![Page 23: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/23.jpg)
Maria Grazia Pia, INFN Genova
Goodness-of-fit
![Page 24: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/24.jpg)
Maria Grazia Pia, INFN Genova
8.1.p01 Jul 2006
9.1 Dec 2007 9.2.p03 Feb 2010
9.3 Dec 2009 9.3 hMS Dec 2009
2200
2300
2400
2500
2600
2700
2800
Geant4 version
To
tal
de
po
sit
ed
en
erg
y (
Ge
V)
2006
Dec.2007
Feb.2010
Dec.2009
Acceptance
99.9% CI
9.3 hMS
9.3 9.2p03 9.1 8.1p02
Total deposited energy
9.3 9.2p03 9.1 8.1p02
9.3 hMS
99.9% CI
8.1p02
9.3 hMS
![Page 25: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/25.jpg)
Maria Grazia Pia, INFN Genova
IEEE Trans. Nucl. Sci., vol. 57, no. 5, pp. 2805-2830,
October 2010
M.G.Pia, M. Begalli, A. Lechner, L. Quintieri, P. Saracco
Physics-related epistemic uncertainties
in protondepth dose simulation
fresh from the oven…
![Page 26: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/26.jpg)
Maria Grazia Pia, INFN Genova
ConclusionsEvaluation of systematic effects associated with epistemic uncertainties Sensitivity analysis (~interval analysis) More refined methods: Dempster-Shafer Methods specific to Monte Carlo simulation?
Complementary statistical methods contribute to identify and quantify effects Qualitative appraisal is not adequate
Epistemic uncertainties are reducible Can be reduced or suppressed by extending knowledge New experimental measurements
Uncertainty quantification gives us guidance about What to measure What experimental precision is needed/adequate Priorities: which uncertainties generate the worst systematic
effects
The impact of epistemic uncertainties depends on
the experimental application environment
![Page 27: Maria Grazia Pia, INFN Genova Epistemic and systematic uncertainties in Monte Carlo simulation: Epistemic and systematic uncertainties in Monte Carlo simulation:](https://reader035.fdocuments.us/reader035/viewer/2022062404/551619d3550346d46f8b6608/html5/thumbnails/27.jpg)
Maria Grazia Pia, INFN Genova
Backup
4.
1 55.
15.
2 6
6.0
-pat
ch1
6.1
6.2
7.1
8.1
8.2 9
9.1
9.2
550
650
750
850
950
1050
1150
1250
1350
1450
100 GeV mu+, 1 m Fe, lateral deviation at end-point
Geant4 version
De
via
tio
n (
mic
rom
)
Geant4/examples/extended/electromagnetic/testEm5/mumsc/deviation.ascii