Gaussian Process Modelling for Uncertainty Quanti cation ...
MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 ·...
Transcript of MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 ·...
![Page 1: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/1.jpg)
MIT Uncertainty Quantification (MUQ) Library
Andrew Davis 1 Matthew Parno 2 Youssef Marzouk 1
1Massachusetts Institute of Technology
2Cold Regions Research and Engineering Lab (CRREL)
SIAM UQ 2016
Davis (MIT) MUQ SIAM UQ 2016 1 / 24
![Page 2: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/2.jpg)
MUQ: philosophy
“A small change in algorithm or application
should only require a small change in code”
Goals:
Provide a sandbox for playing with different models and algorithms
Facilitate constructive modeling
Extensible and easily hybridized algorithms
Reduce distance and time between algorithms and applications
Target audiences:
Algorithm developers
Application scientists and engineers
Davis (MIT) MUQ SIAM UQ 2016 2 / 24
![Page 3: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/3.jpg)
MUQ: philosophy
“A small change in algorithm or application
should only require a small change in code”
Goals:
Provide a sandbox for playing with different models and algorithms
Facilitate constructive modeling
Extensible and easily hybridized algorithms
Reduce distance and time between algorithms and applications
Target audiences:
Algorithm developers
Application scientists and engineers
Davis (MIT) MUQ SIAM UQ 2016 2 / 24
![Page 4: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/4.jpg)
MUQ: outline
1 Modeling framework
Constructing combined (physical + statistical) models with high-level
structure
Graphical modeling technique to easily combine sub models
2 Algorithm framework
Example: Markov chain Monte Carlo stack — using mathematical
structure in software design
3 MUQ development update
Use of MUQ in our research
Davis (MIT) MUQ SIAM UQ 2016 3 / 24
![Page 5: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/5.jpg)
MUQ: modeling
Unique modeling approach:
1 Graphically building physical and statistical models
Connect models and algorithms to expose underlying structure
2 Easy to swap and extend models
Apply MCMC (and other algorithms) to a wide variety of applications
3 “Grey-box” approach: MUQ wraps around existing models, but
uses any new or available problem structure
4 Built-in tools to implement physical models
Davis (MIT) MUQ SIAM UQ 2016 4 / 24
![Page 6: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/6.jpg)
MUQ: modeling
Unique modeling approach:
1 Graphically building physical and statistical models
Connect models and algorithms to expose underlying structure
2 Easy to swap and extend models
Apply MCMC (and other algorithms) to a wide variety of applications
3 “Grey-box” approach: MUQ wraps around existing models, but
uses any new or available problem structure
4 Built-in tools to implement physical models
Davis (MIT) MUQ SIAM UQ 2016 4 / 24
![Page 7: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/7.jpg)
MUQ: modeling
Unique modeling approach:
1 Graphically building physical and statistical models
Connect models and algorithms to expose underlying structure
2 Easy to swap and extend models
Apply MCMC (and other algorithms) to a wide variety of applications
3 “Grey-box” approach: MUQ wraps around existing models, but
uses any new or available problem structure
4 Built-in tools to implement physical models
Davis (MIT) MUQ SIAM UQ 2016 4 / 24
![Page 8: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/8.jpg)
MUQ: modeling
Unique modeling approach:
1 Graphically building physical and statistical models
Connect models and algorithms to expose underlying structure
2 Easy to swap and extend models
Apply MCMC (and other algorithms) to a wide variety of applications
3 “Grey-box” approach: MUQ wraps around existing models, but
uses any new or available problem structure
4 Built-in tools to implement physical models
Davis (MIT) MUQ SIAM UQ 2016 4 / 24
![Page 9: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/9.jpg)
Graphical models in MUQ
Physical & statistical models are
implemented separately (ModPieces)
Each component represents a piece ofthe overall model
(e.g., prior, likelihood, forward model)
Separately implement submodels:
# create submodels
theta = VectorPassthroughModel()
prior = GaussianDensity(mu, sig)
f = ForwardModel()
like = GaussianDensity()
post = DensityProduct()
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 5 / 24
![Page 10: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/10.jpg)
Graphical models in MUQ
Physical & statistical models are
implemented separately (ModPieces)
Each component represents a piece ofthe overall model
(e.g., prior, likelihood, forward model)
Separately implement submodels:
# create submodels
theta = VectorPassthroughModel()
prior = GaussianDensity(mu, sig)
f = ForwardModel()
like = GaussianDensity()
post = DensityProduct()
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 5 / 24
![Page 11: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/11.jpg)
Graphical models in MUQ
Physical & statistical models are
implemented separately (ModPieces)
Each component represents a piece ofthe overall model
(e.g., prior, likelihood, forward model)
Separately implement submodels:
# create submodels
theta = VectorPassthroughModel()
prior = GaussianDensity(mu, sig)
f = ForwardModel()
like = GaussianDensity()
post = DensityProduct()
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 5 / 24
![Page 12: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/12.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Add submodels to the graph:
# Add graph nodes
G = ModGraph()
G.AddNode(theta, ’Parameter’)
G.AddNode(prior, ’Prior’)
G.AddNode(f, ’Forward Model’)
G.AddNode(like, ’Likelihood’)
G.AddNode(post, ’Posterior’)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 6 / 24
![Page 13: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/13.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Add submodels to the graph:
# Add graph nodes
G = ModGraph()
G.AddNode(theta, ’Parameter’)
G.AddNode(prior, ’Prior’)
G.AddNode(f, ’Forward Model’)
G.AddNode(like, ’Likelihood’)
G.AddNode(post, ’Posterior’)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 6 / 24
![Page 14: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/14.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Add submodels to the graph:
# Add graph nodes
G = ModGraph()
G.AddNode(theta, ’Parameter’)
G.AddNode(prior, ’Prior’)
G.AddNode(f, ’Forward Model’)
G.AddNode(like, ’Likelihood’)
G.AddNode(post, ’Posterior’)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 6 / 24
![Page 15: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/15.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Add submodels to the graph:
# Add graph nodes
G = ModGraph()
G.AddNode(theta, ’Parameter’)
G.AddNode(prior, ’Prior’)
G.AddNode(f, ’Forward Model’)
G.AddNode(like, ’Likelihood’)
G.AddNode(post, ’Posterior’)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 6 / 24
![Page 16: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/16.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Add submodels to the graph:
# Add graph nodes
G = ModGraph()
G.AddNode(theta, ’Parameter’)
G.AddNode(prior, ’Prior’)
G.AddNode(f, ’Forward Model’)
G.AddNode(like, ’Likelihood’)
G.AddNode(post, ’Posterior’)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 6 / 24
![Page 17: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/17.jpg)
Graphical models in MUQ
Submodels are connected — becominggraphs (ModGraphs)
Node: A submodel (ModPiece)
Directed edge: out/input of each submodel
Exposes underlying structure
Connect nodes with edges:
# Add graph edges
G.AddEdge(’Parameters’, ’Forward Model’, 0)
G.AddEdge(’Forward Model’, ’Likelihood’, 0)
G.AddEdge(’Likelihood’, ’Posterior’, 0)
G.AddEdge(’Parameters’, ’Prior’, 0)
G.AddEdge(’Prior’, ’Posterior’, 1)
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 7 / 24
![Page 18: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/18.jpg)
Advantages of graph-based modeling
1 Relates implementation structure to
intuitive interpretation
2 Easy to swap pieces of the problem
formulation
# use a different forward model
g = AnotherForwardModel()
G.SwapNode(’Forward Model’, g)
3 “Grey-box” wrapper around existing models
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 8 / 24
![Page 19: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/19.jpg)
Advantages of graph-based modeling
1 Relates implementation structure to
intuitive interpretation
2 Easy to swap pieces of the problem
formulation
# use a different forward model
g = AnotherForwardModel()
G.SwapNode(’Forward Model’, g)
3 “Grey-box” wrapper around existing models
Parameters: θ
Forward model:
g(θ)
Likelihood:
π(d |g(θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 8 / 24
![Page 20: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/20.jpg)
Advantages of graph-based modeling
1 Relates implementation structure to
intuitive interpretation
2 Easy to swap pieces of the problem
formulation
# use a different forward model
g = AnotherForwardModel()
G.SwapNode(’Forward Model’, g)
3 “Grey-box” wrapper around existing models
Parameters: θ
Forward model:
g(θ)
Likelihood:
π(d |g(θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 8 / 24
![Page 21: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/21.jpg)
Grey-box wrapper: Using existing models (C++)
Existing implementation becomes a submodel
The user only needs to write a MUQ/existing solver interface:
class ModelWrapper : public muq::Modeling::ModPiece –
public:
// A way to create the submodel
ModelWrapper(inputSizes, outputSize) :
ModPiece(inputSizes, outputSize)
private:
// Grey-box implementation
Eigen::VectorXd EvaluateImpl(std::vector¡Eigen::VectorXd¿
const& inputs) –
Eigen::VectorXd output(outputSize);
/* ***** CALL EXISTING CODE HERE ***** */
return output;
˝
˝;
Davis (MIT) MUQ SIAM UQ 2016 9 / 24
![Page 22: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/22.jpg)
Grey-box wrapper: Using existing models (Python)
Existing implementation becomes a submodel
The user only needs to write a MUQ/existing solver interface:
class ModelWrapper(libmuqModelling.ModPiece):
# a way to create the submodel
def ˙˙init˙˙(self, inputSizes, outputSize):
libmuqModelling.ModPiece.˙˙init˙˙(inputSizes, outputSize)
# Grey-box implementation
def EvaluateImpl(self, inputs):
# #### CALL EXISTING CODE HERE #### #
# existing code returns output (list type)
return output
Davis (MIT) MUQ SIAM UQ 2016 10 / 24
![Page 23: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/23.jpg)
Derivative implementation (C++)
User may optionally implement derivative informationGradient, Jacobian, Jacobian’s action, and/or Hessian
The user only needs to write a MUQ/existing solver interface:
class ModelWrapper : public muq::Modeling::ModPiece –
// Model implementation
Eigen::VectorXd EvaluateImpl(std::vector¡Eigen::VectorXd¿
const& inputs) – return output; ˝
// Derivative implementation
virtual Eigen::VectorXd
GradientImpl(std::vector¡Eigen::VectorXd¿ const& inputs,
Eigen::VectorXd const& sensitivity,
int const inputDimWrt) override –
return gradient;
˝
˝;
Davis (MIT) MUQ SIAM UQ 2016 11 / 24
![Page 24: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/24.jpg)
Model derivatives
1 Option to include user implemented
derivatives
2 Defaults to finite difference if analytic
derivatives are not implemented
3 Computes derivative of the graph’s output
via the chain rule
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 12 / 24
![Page 25: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/25.jpg)
Model derivatives
1 Option to include user implemented
derivatives
2 Defaults to finite difference if analytic
derivatives are not implemented
3 Computes derivative of the graph’s output
via the chain rule
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 12 / 24
![Page 26: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/26.jpg)
Model derivatives
1 Option to include user implemented
derivatives
2 Defaults to finite difference if analytic
derivatives are not implemented
3 Computes derivative of the graph’s output
via the chain rule
Parameters: θ
Forward model:
f (θ)
Likelihood:
π(d |f (θ))
Prior:
π(θ)
Posterior:
π(θ|d) ∝ π(d |f (θ))π(θ)
Davis (MIT) MUQ SIAM UQ 2016 12 / 24
![Page 27: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/27.jpg)
Built-in modeling tools
1 Extensible submodel class (ModPiece)
2 Implemented common models:
Component-wise functions (exp, sin, . . .)
Component-wise sum
Linear solver Ax = b (using Eigen)
and many more . . .
3 Sundials interface
Numerical integration of ODEs, state-discretized PDEs
Root-finding algorithms
4 Transient and steady PDE solver
User must implement the weak form
Derivative information computed via automatic differentiation
(Sacado)
Finite element method spatial discretization (LibMesh)
Davis (MIT) MUQ SIAM UQ 2016 13 / 24
![Page 28: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/28.jpg)
Built-in modeling tools
1 Extensible submodel class (ModPiece)
2 Implemented common models:
Component-wise functions (exp, sin, . . .)
Component-wise sum
Linear solver Ax = b (using Eigen)
and many more . . .
3 Sundials interface
Numerical integration of ODEs, state-discretized PDEs
Root-finding algorithms
4 Transient and steady PDE solver
User must implement the weak form
Derivative information computed via automatic differentiation
(Sacado)
Finite element method spatial discretization (LibMesh)
Davis (MIT) MUQ SIAM UQ 2016 13 / 24
![Page 29: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/29.jpg)
Built-in modeling tools
1 Extensible submodel class (ModPiece)
2 Implemented common models:
Component-wise functions (exp, sin, . . .)
Component-wise sum
Linear solver Ax = b (using Eigen)
and many more . . .
3 Sundials interface
Numerical integration of ODEs, state-discretized PDEs
Root-finding algorithms
4 Transient and steady PDE solver
User must implement the weak form
Derivative information computed via automatic differentiation
(Sacado)
Finite element method spatial discretization (LibMesh)
Davis (MIT) MUQ SIAM UQ 2016 13 / 24
![Page 30: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/30.jpg)
Built-in modeling tools
1 Extensible submodel class (ModPiece)
2 Implemented common models:
Component-wise functions (exp, sin, . . .)
Component-wise sum
Linear solver Ax = b (using Eigen)
and many more . . .
3 Sundials interface
Numerical integration of ODEs, state-discretized PDEs
Root-finding algorithms
4 Transient and steady PDE solver
User must implement the weak form
Derivative information computed via automatic differentiation
(Sacado)
Finite element method spatial discretization (LibMesh)
Davis (MIT) MUQ SIAM UQ 2016 13 / 24
![Page 31: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/31.jpg)
MUQ: outline
1 Modeling framework
Constructing combined (physical + statistical) models with high-level
structure
Graphical modeling technique to easily combine sub models
2 Algorithm framework
Example: Markov chain Monte Carlo stack — using mathematical
structure in software design
3 MUQ development update
Use of MUQ in our research
Davis (MIT) MUQ SIAM UQ 2016 14 / 24
![Page 32: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/32.jpg)
MUQ: MCMC
Question
What are the fundamental components of an MCMC algorithm?
Metropolis-Hastings MCMC:
1 Save current state
θ(k) = θ
2 Propose new state
θ′ ∼ q(θ′|θ)
3 Accept or reject
α = min
{1,π(θ′)
π(θ)
q(θ|θ′)q(θ′|θ)
}
Chain Kernel Proposal
We copy this structure with
C++ classes
Easily extendable
Many possible configurations
(> 30)
Davis (MIT) MUQ SIAM UQ 2016 15 / 24
![Page 33: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/33.jpg)
MUQ: MCMC
Question
What are the fundamental components of an MCMC algorithm?
Metropolis-Hastings MCMC:
1 Save current state
θ(k) = θ
2 Propose new state
θ′ ∼ q(θ′|θ)
3 Accept or reject
α = min
{1,π(θ′)
π(θ)
q(θ|θ′)q(θ′|θ)
}
Chain Kernel Proposal
We copy this structure with
C++ classes
Easily extendable
Many possible configurations
(> 30)
Davis (MIT) MUQ SIAM UQ 2016 15 / 24
![Page 34: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/34.jpg)
MUQ: MCMC
Question
What are the fundamental components of an MCMC algorithm?
Metropolis-Hastings MCMC:
1 Save current state
θ(k) = θ
2 Propose new state
θ′ ∼ q(θ′|θ)
3 Accept or reject
α = min
{1,π(θ′)
π(θ)
q(θ|θ′)q(θ′|θ)
}
Chain Kernel Proposal
We copy this structure with
C++ classes
Easily extendable
Many possible configurations
(> 30)
Davis (MIT) MUQ SIAM UQ 2016 15 / 24
![Page 35: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/35.jpg)
MUQ: full MCMC framework
Chain Kernel Proposal
AMALA
AM
RW
smMALA
HMC
MH
DR
TM
IA
Single
Multi
Advantages:
New proposals used with any kernel
Focus efforts on new features
Changing components is trivial
Extensive performance comparisons
are straightforward
// MH + AMALA:
ps.put(”MCMC.Kernel”,”MH”);
ps.put(”MCMC.Proposal”,”AMALA”);
// TM + RW:
ps.put(”MCMC.Kernel”,”TransportMap”);
ps.put(”MCMC.Proposal”,”MHProposal”);
Davis (MIT) MUQ SIAM UQ 2016 16 / 24
![Page 36: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/36.jpg)
MCMC and models
Observation:
Many MCMC algorithms rely on specific problem structure.
Solution:
Use problem classes that can exploit graphical representations.
MCMC
Algorithm
Sampling
Problem
Graphical
Model
Problem iterates through graph and extracts important structure:
Gaussian prior and error model
Linear models
Gauss-Newton Hessians
etc. . .
No extra effort: Structure automatically defined in “grey-box” model.
Davis (MIT) MUQ SIAM UQ 2016 17 / 24
![Page 37: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/37.jpg)
MCMC and models
Observation:
Many MCMC algorithms rely on specific problem structure.
Solution:
Use problem classes that can exploit graphical representations.
MCMC
Algorithm
Sampling
Problem
Graphical
Model
Problem iterates through graph and extracts important structure:
Gaussian prior and error model
Linear models
Gauss-Newton Hessians
etc. . .
No extra effort: Structure automatically defined in “grey-box” model.
Davis (MIT) MUQ SIAM UQ 2016 17 / 24
![Page 38: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/38.jpg)
MCMC and models
Observation:
Many MCMC algorithms rely on specific problem structure.
Solution:
Use problem classes that can exploit graphical representations.
MCMC
Algorithm
Sampling
Problem
Graphical
Model
Problem iterates through graph and extracts important structure:
Gaussian prior and error model
Linear models
Gauss-Newton Hessians
etc. . .
No extra effort: Structure automatically defined in “grey-box” model.
Davis (MIT) MUQ SIAM UQ 2016 17 / 24
![Page 39: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/39.jpg)
Beyond MCMC
Lessons learned:
Design code to mimic the mathematics
Allows us easily add new components without duplicate code
Makes algorithms more configurable and intuitive
Use gray-box for automated structure extraction
Enables algorithms to exploit high-level structure
Still allows arbitrary model implementations (e.g., legacy code)
Notes:
Similar structure exists in our optimization algorithms
More complicated chains being tested (DILI, DILI+maps, etc.)
Davis (MIT) MUQ SIAM UQ 2016 18 / 24
![Page 40: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/40.jpg)
Beyond MCMC
Lessons learned:
Design code to mimic the mathematics
Allows us easily add new components without duplicate code
Makes algorithms more configurable and intuitive
Use gray-box for automated structure extraction
Enables algorithms to exploit high-level structure
Still allows arbitrary model implementations (e.g., legacy code)
Notes:
Similar structure exists in our optimization algorithms
More complicated chains being tested (DILI, DILI+maps, etc.)
Davis (MIT) MUQ SIAM UQ 2016 18 / 24
![Page 41: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/41.jpg)
Beyond MCMC
Lessons learned:
Design code to mimic the mathematics
Allows us easily add new components without duplicate code
Makes algorithms more configurable and intuitive
Use gray-box for automated structure extraction
Enables algorithms to exploit high-level structure
Still allows arbitrary model implementations (e.g., legacy code)
Notes:
Similar structure exists in our optimization algorithms
More complicated chains being tested (DILI, DILI+maps, etc.)
Davis (MIT) MUQ SIAM UQ 2016 18 / 24
![Page 42: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/42.jpg)
MUQ: outline
1 Modeling framework
Constructing combined (physical + statistical) models with high-level
structure
Graphical modeling technique to easily combine sub models
2 Algorithm framework
Example: Markov chain Monte Carlo stack — using mathematical
structure in software design
3 MUQ development update
Use of MUQ in our research
Davis (MIT) MUQ SIAM UQ 2016 19 / 24
![Page 43: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/43.jpg)
MUQ usage: ice sheet modeling
Solve conservation equations with uncertain input parameters . . .
Uncertain basal topography
Uncertain basal friction Uncertain width
Longitudinal
stress− Frictional
stress− Lateral
stress= Driving
stress
Momentum balance equation
Change in
thickness+ Advection = Accumulation
and ablation
Stochastic
calvingMass balance equation
. . . computed field are used within UQ algorithms (e.g., Monte Carlo,
optimization, MCMC, importance sampling ect. . . .)Davis (MIT) MUQ SIAM UQ 2016 20 / 24
![Page 44: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/44.jpg)
MUQ usage: map-accelerated MCMC
Use nonlinear variable transformation to “precondition” target density.
reference proposal
qr (r′|r) = N(r , σ2I )
mapped proposal
qθ(θ′|θ) = qr
(T̃ (θ′)|T̃ (θ)
) ∣∣detDT̃ (θ′)∣∣
The exchangeability of MCMC components makes testing many
reference proposals easy
Davis (MIT) MUQ SIAM UQ 2016 21 / 24
![Page 45: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/45.jpg)
MUQ usage: characterizing marginal distributions
Build a surrogate model of the marginal using stochastic evaluations
Using the surrogate to evaluate the acceptance ratio does not affect
the proposal
We can easily combine many proposal methods (e.g., MH, AM,
DRAM, . . .) with surrogate modeling
−2 −1 0 1 2 3 4 5
θ
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
π(θ
)
Davis (MIT) MUQ SIAM UQ 2016 22 / 24
![Page 46: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/46.jpg)
MUQ usage: characterizing marginal distributions
Build a surrogate model of the marginal using stochastic evaluations
Using the surrogate to evaluate the acceptance ratio does not affect
the proposal
We can easily combine many proposal methods (e.g., MH, AM,
DRAM, . . .) with surrogate modeling
−2 −1 0 1 2 3 4 5
θ
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
π(θ
)
Davis (MIT) MUQ SIAM UQ 2016 22 / 24
![Page 47: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/47.jpg)
MUQ usage: characterizing marginal distributions
Build a surrogate model of the marginal using stochastic evaluations
Using the surrogate to evaluate the acceptance ratio does not affect
the proposal
We can easily combine many proposal methods (e.g., MH, AM,
DRAM, . . .) with surrogate modeling
−2 −1 0 1 2 3 4 5
θ
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
π(θ
)
Davis (MIT) MUQ SIAM UQ 2016 22 / 24
![Page 48: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/48.jpg)
MUQ: development notes
Development improvements:
Improved build systemcmake automatically downloads and installs dependencies
Extensive testing for corrent and/or consistent perfermanceAutomated code testing for verification with Jenkins
Google analytics:
0 2 4 6 8 10 12 14 16 180
50
100Workshop SIAM CSE
Weeks since 1 Dec 2014
Ses
sio
ns/
wee
k BitBucket
Doxygen
Davis (MIT) MUQ SIAM UQ 2016 23 / 24
![Page 49: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/49.jpg)
MUQ: development notes
Development improvements:Improved build system
cmake automatically downloads and installs dependencies
Extensive testing for corrent and/or consistent perfermanceAutomated code testing for verification with Jenkins
Google analytics:
0 2 4 6 8 10 12 14 16 180
50
100Workshop SIAM CSE
Weeks since 1 Dec 2014
Ses
sio
ns/
wee
k BitBucket
Doxygen
Davis (MIT) MUQ SIAM UQ 2016 23 / 24
![Page 50: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/50.jpg)
MUQ: development notes
Development improvements:Improved build system
cmake automatically downloads and installs dependencies
Extensive testing for corrent and/or consistent perfermanceAutomated code testing for verification with Jenkins
Google analytics:
0 2 4 6 8 10 12 14 16 180
50
100Workshop SIAM CSE
Weeks since 1 Dec 2014
Ses
sio
ns/
wee
k BitBucket
Doxygen
Davis (MIT) MUQ SIAM UQ 2016 23 / 24
![Page 51: MIT Uncertainty Quanti cation (MUQ) Libraryneckel/siamuq16_slides_minisymp/... · 2016-04-28 · MIT Uncertainty Quanti cation (MUQ) Library Andrew Davis 1 Matthew Parno 2 Youssef](https://reader034.fdocuments.us/reader034/viewer/2022042206/5ea807220e6d2006700c41fc/html5/thumbnails/51.jpg)
muq.mit.edu
Davis (MIT) MUQ SIAM UQ 2016 24 / 24