Monte Carlo Methods for Uncertainty Quantification
Abdul-Lateef Haji-AliBased on slides by: Mike Giles
Mathematical Institute, University of Oxford
Contemporary Numerical Techniques
Haji-Ali (Oxford) Monte Carlo methods 1 / 27
Lecture outline
Lecture 4: PDE applications
PDEs with uncertainty
examples
multilevel Monte Carlo
Extensions
Haji-Ali (Oxford) Monte Carlo methods 2 / 27
PDEs with Uncertainty
Looking at the history of numerical methods for PDEs, the first stepswere about improving the modelling:
1D → 2D → 3D
steady → unsteady
laminar flow → turbulence modelling → large eddy simulation→ direct Navier-Stokes
simple geometries (e.g. a wing) → complex geometries(e.g. an aircraft in landing configuration)
adding new features such as combustion, coupling to structural /thermal analyses, etc.
. . . and then engineering switched from analysis to design.
Haji-Ali (Oxford) Monte Carlo methods 3 / 27
PDEs with Uncertainty
The big move now is towards handling uncertainty:
uncertainty in modelling parameters
uncertainty in geometry
uncertainty in initial conditions
uncertainty in spatially-varying material properties
inclusion of stochastic source terms
Engineering wants to move to “robust design” taking into account theeffects of uncertainty.
Other areas want to move into Bayesian inference, starting with an a prioridistribution for the uncertainty, and then using data to derive an improveda posteriori distribution.
Haji-Ali (Oxford) Monte Carlo methods 4 / 27
PDEs with Uncertainty
Range of applications: parabolic, elliptic, hyperbolic, numericalhomogenization, reduced basis approximation.Examples:
Long-term climate modelling:
Lots of sources of uncertainty including the effects of aerosols,clouds, carbon cycle, ocean circulation(http://climate.nasa.gov/uncertainties)
Short-range weather prediction
Considerable uncertainty in the initial data due to limitedmeasurements
Haji-Ali (Oxford) Monte Carlo methods 5 / 27
PDEs with Uncertainty
Engineering analysis
Perhaps the biggest uncertainty is geometric due to manufacturingtolerances
Nuclear waste repository and oil reservoir modelling
Considerable uncertainty about porosity of rock
Astronomy
“Random” spatial/temporal variations in air density disturbcorrelation in signals received by different antennas
Finance
Stochastic forcing due to market behaviour
Haji-Ali (Oxford) Monte Carlo methods 6 / 27
PDEs with Uncertainty
In the past, Monte Carlo simulation has been viewed as impractical due toits expense, and so people have used other methods:
stochastic collocation
polynomial chaos
Because of Multilevel Monte Carlo, this is changing and there are nowseveral research groups using MLMC for PDE applications
The approach is very simple, in principle:
use a sequence of grids of increasing resolution in space (and time)
as with SDEs, determine the optimal allocation of computationaleffort on the different levels
the savings can be much greater because the cost goes up morerapidly with level (higher dimensionality).
Haji-Ali (Oxford) Monte Carlo methods 7 / 27
PDEs with Uncertainty: Example
Elliptic PDE coming from Darcy’s law:
∇·(κ(x)∇p
)= 0
where the permeability κ(x) is uncertain, and log κ(x) is often modelled asbeing Normally distributed with a spatial covariance such as
R(log κ(x1), log κ(x2)) = σ2 exp(−‖x1−x2‖/λ)
Modelling of oil reservoirs and groundwater contamination in nuclearwaste repositories (Giles, Scheichl, Cliffe, 2011).
Haji-Ali (Oxford) Monte Carlo methods 8 / 27
Stochastic Field
A typical realisation of κ for λ = 0.01, σ = 1.
Haji-Ali (Oxford) Monte Carlo methods 9 / 27
Stochastic Field
Samples of log κ are provided by a Karhunen-Loeve expansion:
log κ(x , ω) =∞∑n=0
√θn ξn(ω) fn(x),
where θn, fn are eigenvalues / eigenfunctions of the correlation function:∫R(x , y) fn(y) dy = θn fn(x)
and ξn(ω) are standard Normal random variables.
Numerical experiments truncate the expansion.
(Latest 2D/3D work uses an efficient FFT construction based on acirculate embedding.)
Haji-Ali (Oxford) Monte Carlo methods 10 / 27
Stochastic FieldDecay of 1D eigenvalues
100
101
102
103
10−6
10−4
10−2
100
n
eig
en
va
lue
λ=0.01
λ=0.1
λ=1
When λ = 1, can use a low-dimensional polynomial chaos approach, butit’s impractical for smaller λ.
Haji-Ali (Oxford) Monte Carlo methods 11 / 27
PDEs with Uncertainty: Results
Boundary conditions for unit square [0, 1]2:– fixed pressure: p(0, x2)=1, p(1, x2)=0– Neumann b.c.: ∂p/∂x2(x1, 0)=∂p/∂x2(x1, 1)=0
Output quantity – mass flux: −∫
k∂p
∂x1dx2
Correlation length: λ = 0.2
Coarsest grid: h = 1/8 (comparable to λ)
Finest grid: h = 1/128
Karhunen-Loeve truncation: mKL = 4000
Cost taken to be proportional to number of nodes. ∼ h2`
Haji-Ali (Oxford) Monte Carlo methods 12 / 27
2D Results
0 1 2 3 4−12
−10
−8
−6
−4
−2
0
2
level l
log
2 v
aria
nce
Pl
Pl− P
l−1
0 1 2 3 4−12
−10
−8
−6
−4
−2
0
2
level l
log
2 |m
ea
n|
Pl
Pl− P
l−1
V[P`−P`−1] ∼ h2` E[P`−P`−1] ∼ h2`
Haji-Ali (Oxford) Monte Carlo methods 13 / 27
MLMC General Theorem
If there exist independent estimators Y` based on N` Monte Carlo samples,each costing C`, and positive constants α, β, γ, c1, c2, c3 such thatα≥ 1
2 min(β, γ) and
i)∣∣∣E[P`−P]
∣∣∣ ≤ c1 2−α `
ii) E[Y`] =
E[P0], ` = 0
E[P`−P`−1], ` > 0
iii) V[Y`] ≤ c2N−1` 2−β `
iv) E[C`] ≤ c3 2γ `
Haji-Ali (Oxford) Monte Carlo methods 14 / 27
MLMC General Theorem
then there exists a positive constant c4 such that for any ε<1 there existL and N` for which the multilevel estimator
Y =L∑`=0
Y`,
has a mean-square-error with bound E[(
Y − E[P])2]
< ε2
with a computational cost C with bound
C ≤
c4 ε−2, β > γ,
c4 ε−2(log ε)2, β = γ,
c4 ε−2−(γ−β)/α, 0 < β < γ.
Haji-Ali (Oxford) Monte Carlo methods 15 / 27
2D Results
0 1 2 3 410
2
103
104
105
106
107
108
level l
Nl
10−3
10−2
100
101
102
accuracy ε
ε2 C
ost
Std MC
MLMCε=0.0005
ε=0.001
ε=0.002
ε=0.005
ε=0.01
Haji-Ali (Oxford) Monte Carlo methods 16 / 27
Complexity analysis
Relating things back to the MLMC theorem:
E[P`−P] ∼ 2−2` =⇒ α = 2
V` ∼ 2−2` =⇒ β = 2
C` ∼ 2d` =⇒ γ = d (dimension of PDE)
To achieve r.m.s. accuracy ε requires finest level grid spacing h ∼ ε1/2and hence we get the following complexity:
dim MC MLMC
1 ε−2.5 ε−2
2 ε−3 ε−2(log ε)2
3 ε−3.5 ε−2.5
Haji-Ali (Oxford) Monte Carlo methods 17 / 27
Non-geometric multilevel
Almost all applications of multilevel in the literature so far use a geometricsequence of levels, refining the timestep (or the spatial discretisation forPDEs) by a constant factor when going from level ` to level `+ 1.
Coming from a multigrid background, this is very natural, but it is NOTa requirement of the multilevel Monte Carlo approach.
All MLMC needs is a sequence of levels with
increasing accuracy
increasing cost
increasingly small difference between outputs on successive levels
Haji-Ali (Oxford) Monte Carlo methods 18 / 27
Applying MLMC to your problems
First, identify the approximation parameter(s) in the problem to builda hierarchy of approximations. We also need to make sure that wecan to sample correlated approximations.
Try to find estimators that increase the variance convergence rate, β,as much as possible to get the optimal complexity.
The MLMC/MIMC theory depends on a set of assumptions.Checking these assumptions numerically is straightforward butproving them can be challenging.
Haji-Ali (Oxford) Monte Carlo methods 19 / 27
Multilevel Monte Carlo: Summary
Let ∆`P = P` − P`−1 and ∆0 = P0 and observe that
E[P] =∞∑`=0
E[∆`P] ≈L∑`=0
E[∆`P].
Then, the MLMC estimator
L∑`=0
1
N`
N∑m=1
∆`P(`,m)`
with optimal choice of L and N` has complexity
O(ε−2−max(0, γ−βα )
),
when γ 6= β and O(ε−2|log ε|2
)otherwise.
Haji-Ali (Oxford) Monte Carlo methods 20 / 27
Multilevel Monte Carlo for high-dimensional problems
The problem is that the cost usually grows exponentially as the number ofdiscretization parameters, d , increases, i.e., the cost is O
(2dγ`
).
On the other hand, the weak error, O(2−α`
), and variance, O
(2−β`
), do
not decrease with increasing number of discretization parameters.
Leading to a complexity that suffers from the curse of dimensionality
O(ε−2−max(0, dγ−βα )
).
Can we leverage some mixed-regularity between these discretizationparameters to increase the convergence rate of the weak error and thevariance?
Haji-Ali (Oxford) Monte Carlo methods 21 / 27
Multi-Index Monte Carlo
Haji-Ali (Oxford) Monte Carlo methods 22 / 27
Multi-Index Monte Carlo
Haji-Ali (Oxford) Monte Carlo methods 22 / 27
Multi-Index Monte CarloLet ` = (`i )
di=1 ∈ Nd be a “multi-index” for d discretization parameters
and denote by P` the corresponding approximation.Then, define the i th-dimension difference operator
∆i ,`P =
{P` if `i = 0,
P` − P`−e iotherwise
and ∆`P =d⊗
i=1
∆i ,`P.
Observe, again, that
E[g ] =∑`∈Nd
E[∆`P] ≈∑`∈I
E[∆`P],
for some I ⊂ Nd . Then, estimate the expectations with independentMonte Carlo samplers to yield the MIMC estimator
∑`∈I
1
N`
N∑m=1
∆`P(m).
Haji-Ali (Oxford) Monte Carlo methods 23 / 27
Multi-Index Monte Carlo
Assuming that the cost is O(2γ|`|
),
|E[∆`g ]| = O(
2−α|`|),
and V[∆`g ] = O(
2−β|`|),
Decreasing ε
`1
`2
where |`| = `1 + `2 + . . . `d . Then, there is an optimal choice of N` and I(total degree or simplex) so that the complexity of MIMC is
O(ε−2−max(0, γ−βα )|log ε|d−1
),
when γ 6= β and O(ε−2|log ε|2d+max(0,d−3)
)otherwise.
Haji-Ali (Oxford) Monte Carlo methods 24 / 27
Other MLMC extensions
Unbiased MLMC: Based on the observation
E[P] =∞∑`=0
E[∆`P] = E[∆`P/p`],
where, in the RHS, ` is a random variable with P[` = i ] = pi , chosenoptimally.Useful when β > γ to simplify the analysis of some compositemethods but has the same complexity as MLMC.
Multilevel Richardson-Romberg extrapolation (ML2R): ApplyingRichardson extrapolation to the weak error estimate, find optimal w`in
E[PL] =L∑`=0
w`E[∆`P]
to have a smaller number of required levels, L. ML2R is useful whenβ ≤ γ and improves the computational complexity.
Haji-Ali (Oxford) Monte Carlo methods 25 / 27
Other hierarchical samplers
The same multilevel/multi-index ideas (and extensions) can be easilyapplied (and laboriously analysed) to other samplers such as:
Stochastic Collocation and Quasi-Monte Carlo: When we have highermixed regularity between the stochastic variables. Leads to bettercomplexities.
Random L2 projection: To produce surrogate models of the quantitiesof interest instead of computing their expectations. Useful forregression and Machine Learning.
Markov Chain Monte Carlo or Sequential Monte Carlo: for inverseproblems where the expectation is with respect to a posteriordistribution depending on a set of observed data points.
Haji-Ali (Oxford) Monte Carlo methods 26 / 27
Final comments
Uncertainty Quantification is a hot topic, with its own conferencesand journals
Monte Carlo methods are a powerful approach to handle uncertaintyin a number of different settings
Multilevel Monte Carlo greatly reduces the cost in a lot of settings,particularly when dealing with PDEs
for more details, see (Giles, 2015) in Acta Numerica.
Haji-Ali (Oxford) Monte Carlo methods 27 / 27
Top Related