Machine Learning for Inverse Problems in Computational ...
Transcript of Machine Learning for Inverse Problems in Computational ...
Machine Learning for Inverse Problems inComputational Engineering
Kailai Xu, and Eric Darvehttps://github.com/kailaix/ADCME.jl
ADCME ML for Computational Engineering 1 / 1
Outline
ADCME ML for Computational Engineering 2 / 1
Inverse Modeling
Inverse modeling identifies a certain set of parameters or functionswith which the outputs of the forward analysis matches the desiredresult or measurement.
Many real life engineering problems can be formulated as inversemodeling problems: shape optimization for improving the performanceof structures, optimal control of fluid dynamic systems, etc.
Physical Properties
Physical Laws
Predictions (Observations)
Inverse Modeling
……
Optimal Control
Predictive Modeling
Discover Physics
Reduced Order Modeling
ADCME ML for Computational Engineering 3 / 1
Inverse Modeling
Forward Problem
Inverse Problem
Model Parameters
Observations
Physical Laws
Physical LawsEstimation
of Parameters
Prediction of
Observations
ADCME ML for Computational Engineering 4 / 1
Inverse Modeling
We can formulate inverse modeling as a PDE-constrained optimizationproblem
minθ
Lh(uh) s.t. Fh(θ, uh) = 0
The loss function Lh measures the discrepancy between the predictionuh and the observation uobs, e.g., Lh(uh) = ‖uh − uobs‖2
2.
θ is the model parameter to be calibrated.
The physics constraints Fh(θ, uh) = 0 are described by a system ofpartial differential equations. Solving for uh may require solving linearsystems or applying an iterative algorithm such as theNewton-Raphson method.
ADCME ML for Computational Engineering 5 / 1
Function Inverse Problem
minf
Lh(uh) s.t. Fh(f , uh) = 0
What if the unknown is a function instead of a set of parameters?
Koopman operator in dynamical systems.
Constitutive relations in solid mechanics.
Turbulent closure relations in fluid mechanics.
...
The candidate solution space is infinite dimensional.
ADCME ML for Computational Engineering 6 / 1
Machine Learning for Computational Engineering
minθ
Lh(uh) s.t. Fh(NNθ, uh) = 0
Deep neural networks exhibit capability of approximating highdimensional and complicated functions.Machine Learning for Computational Engineering: the unknownfunction is approximated by a deep neural network, and the physicalconstraints are enforced by numerical schemes.Satisfy the physics to the largest extent.
ADCME ML for Computational Engineering 7 / 1
Gradient Based Optimization
minθ
Lh(uh) s.t. Fh(θ, uh) = 0 (1)
We can now apply a gradient-based optimization method to (??).The key is to calculate the gradient descent direction gk
θk+1 ← θk − αgk
Predicted Data
Observed Data
Loss Function
Calculate Gradients
Update Model Parameters
PDE
Initial and Boundary Conditions
Optimizer
< tol?Calibrated Model
ADCME ML for Computational Engineering 8 / 1
Outline
ADCME ML for Computational Engineering 9 / 1
Automatic Differentiation
The fact that bridges the technical gap between machine learning andinverse modeling:
Deep learning (and many other machine learning techniques) andnumerical schemes share the same computational model: compositionof individual operators.
Mathematical Fact
Back-propagation||
Reverse-modeAutomatic Differentiation
||Discrete
Adjoint-State Method
ADCME ML for Computational Engineering 10 / 1
Computational Graph for Numerical Schemes
To leverage automatic differentiation for inverse modeling, we need toexpress the numerical schemes in the “AD language”: computationalgraph.
No matter how complicated a numerical scheme is, it can bedecomposed into a collection of operators that are interlinked viastate variable dependencies.
S2
u ϕ
mtΨ2
ϕ(Sn+12 − Sn2) − ∇ ⋅ (m2(Sn+1
2 )K ∇Ψn2) Δt = (qn2 + qn1m2(Sn+12 )m1(Sn+12 ) ) Δt
S2
u ϕ
mtΨ2
S2
u ϕ
mtΨ2
tn tn+1 tn+2
ADCME ML for Computational Engineering 11 / 1
ADCME: Computational-Graph-based NumericalSimulation
ADCME ML for Computational Engineering 12 / 1
Distributed Optimization
ADCME also supports MPI-based distributed computing. The parallelmodel is designed specially for scientific computing.
Key idea: Everything is an operator. Computation andcommunications are converters of data streams (tensors) through thecomputational graph.
mpi bcast, mpi sum, mpi send, mpi recv, mpi halo exchange, ...
ADCME ML for Computational Engineering 13 / 1
Automatic Differentiation: Forward-mode andReverse-mode
ADCME ML for Computational Engineering 14 / 1
What is the Appropriate Model for Inverse Problems?
In general, for a function f : Rn → Rm
Mode Suitable for ... Complexity1 Application
Forward m n ≤ 2.5 OPS(f (x)) UQReverse m n ≤ 4 OPS(f (x)) Inverse Modeling
There are also many other interesting topics
Mixed mode AD: many-to-many mappings.Computing sparse Jacobian matrices using AD by exploiting sparsestructures.
Margossian CC. A review of automatic differentiation and its efficient implementation. Wiley
Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 2019 Jul;9(4):e1305.
1OPS is a metric for complexity in terms of fused-multiply adds.ADCME ML for Computational Engineering 15 / 1
Granularity of Automatic Differentiation
ADCME ML for Computational Engineering 16 / 1
Outline
ADCME ML for Computational Engineering 17 / 1
Inverse Modeling of the Stokes Equation
The governing equation for the Stokes problem
−ν∆u +∇p = f in Ω
∇ · u = 0 in Ω
u = 0 on ∂Ω
The weak form is given by
(ν∇u,∇v)− (p,∇ · v) = (f , v)
(∇ · u, q) = 0
ADCME ML for Computational Engineering 18 / 1
Inverse Modeling of the Stokes Equation
nu = Variable(0.5)
K = nu*constant(compute fem laplace matrix(m, n, h))
B = constant(compute interaction matrix(m, n, h))
Z = [K -B'
-B spdiag(zeros(size(B,1)))]
# Impose boundary conditions
bd = bcnode("all", m, n, h)
bd = [bd; bd .+ (m+1)*(n+1); ((1:m) .+ 2(m+1)*(n+1))]
Z, _ = fem impose Dirichlet boundary condition1(Z, bd, m, n, h)
# Calculate the source term
F1 = eval f on gauss pts(f1func, m, n, h)
F2 = eval f on gauss pts(f2func, m, n, h)
F = compute fem source term(F1, F2, m, n, h)
rhs = [F;zeros(m*n)]
rhs[bd] .= 0.0
sol = Z\rhsADCME ML for Computational Engineering 19 / 1
Inverse Modeling of the Stokes Equation
The distinguished feature compared to traditional forward simulationprograms: the model output is differentiable with respect to modelparameters!
loss = sum((sol[idx] - observation[idx])^2)
g = gradients(loss, nu)
Optimization with a one-liner:
BFGS!(sess, loss)
ADCME ML for Computational Engineering 20 / 1
Outline
ADCME ML for Computational Engineering 21 / 1
Learning spatially-varying physical parameters using deepneural networks
It is easy to adopt ADCME for modeling spatially-varying physicalparameters using deep neural networks with a PDE solver.
DNN + PDE + Data = Physics Constrained Data-driven Modeling
Deep Neural Network PDE Solver Observation
Loss Function
Spatially-varying Physical Fields
Gradient Back-propagation
Forward Computation
ADCME ML for Computational Engineering 22 / 1
Linear Elasticity
DNN + Linear Elasticity + Displacement Data
σij ,j + bi = 0, x ∈ Ω
εij =1
2(uj ,i + ui ,j), x ∈ Ω
σij = λδijεkk + µ(εij + εji ), x ∈ Ω
σijnj = tj , x ∈ ΓN ; ui = (u0)i , x ∈ ΓD
λ =Eν
(1 + ν)(1− 2ν)µ =
Eν
1− ν2
figures/linear_elasticity.png
ADCME ML for Computational Engineering 23 / 1
Stokes’ Problem
DNN + Stokes’ Problem + Pressure Data
−∇ · (ν∇u) +∇p = f in Ω
∇ · u = 0 in Ω
u = 0 on ∂Ω
figures/stokes_problem.png
ADCME ML for Computational Engineering 24 / 1
Hyperelasticity
DNN + Hyperelasticity + Displacement Data
minuψ =
µ
2(Ic − 2)− µ
2log(J) +
λ
8log(J)2
F =I +∇u, C = FTF , J = det(C ), Ic = trace(C )
λ =Eν
(1 + ν)(1− 2ν), µ =
E
2(1 + ν)
figures/hyperelasticity.png
ADCME ML for Computational Engineering 25 / 1
Burgers’ Equation
DNN + Burgers’ Equation + Velocity Data
∂u
∂t+ u
∂u
∂x+ v
∂u
∂y= ∇ · (ν∇u)
∂v
∂t+ u
∂v
∂x+ v
∂v
∂y= ∇ · (ν∇v)
(x , y) ∈ Ω, t ∈ (0,T )
figures/burgers.png
ADCME ML for Computational Engineering 26 / 1
Navier-Stokes Equation
Steady-state Navier-Stokes equation
(u · ∇)u = −1
ρ∇p +∇ · (ν∇u) + g
∇ · u = 0
Inverse problem are ubiquitous in fluid dynamics:
Figure: Left: electronic cooling; right: nasal drug delivery.
ADCME ML for Computational Engineering 27 / 1
Navier-Stokes Equation
BoundaryConditions
Neural NetworkWeights and Biases
Neural Network
Viscosity
Coordinates
𝑢𝑛+1 = 𝑢𝑛 − 𝐽−1𝐹𝑛Newton’s Iteration
𝑢1
𝑢2
𝑢3
𝑢4
Loss Function
Forward Computation
Gradient Backpropagation
ADCME ML for Computational Engineering 28 / 1
Navier-Stokes Equation
Data: (u, v)Unknown: ν(x) (represented by a deep neural network)Prediction: p (absent in the training data)The DNN provides regularization, which generalizes the estimationbetter!
ADCME ML for Computational Engineering 29 / 1
ADSeismic.jl: A General Approach to Seismic Inversion
Many seismic inversion problems can be solved within a unifiedframework.
ADCME ML for Computational Engineering 30 / 1
NNFWI: Neural-network-based Full-Waveform Inversion
Estimate velocity models from seismic observations.
∂2u
∂t2= ∇·(m2∇u)+f
ADCME ML for Computational Engineering 31 / 1
NNFWI: Neural-network-based Full-Waveform Inversion
Inversion results with a noise level σ = σ0
Inversion results for the same loss function value:
ADCME ML for Computational Engineering 32 / 1
ADSeismic.jl: Performance Benchmark
Performance is a key focus of ADCME.ADCME enables us to utilize heterogeneous (CPUs, GPUs, andTPUs) and distributed (CPU clusters) computing environments.Fortran: open-source Fortran90 programs SEISMIC CPML
ADCME ML for Computational Engineering 33 / 1
Constitutive Modeling
ADCME ML for Computational Engineering 34 / 1
Poroelasticity
Multi-physics Interaction of Coupled Geomechanics and Multi-PhaseFlow Equations
divσ(u)− b∇p = 0
1
M
∂p
∂t+ b
∂εv (u)
∂t−∇ ·
(k
Bf µ∇p
)= f (x , t)
σ = σ(ε, ε)
Approximate the constitutive relation by a neural network
σn+1 = H(εn+1 − εn) +NN θ(σn, εn)
Traction-free∂u∂n
= 0
No-flow∂p∂n
= 0
Fixed Pressurep = 0
No-flow∂p∂n
= 0No-flow∂p∂n
= 0 Injection Production
x
y
Finite Element Finite Volume Cell
He1 He2
He3 He4
e
Sensors
ADCME ML for Computational Engineering 35 / 1
Poroelasticity
Comparison with space varying linear elasticity approximation
σ = H(x , y)ε
ADCME ML for Computational Engineering 36 / 1
Poroelasticity
ADCME ML for Computational Engineering 37 / 1
A Paradigm for Inverse Modeling
Most inverse modeling problems can be classified into 4 categories.To be more concrete, consider the PDE for describing physics
∇ · (θ∇u(x)) = 0 BC(u(x)) = 0 (2)
We observe some quantities depending on the solution u and want toestimate θ.
Expression Description ADCME Solution Note
∇ · (c∇u(x)) = 0 Parameter Inverse ProblemDiscrete Adjoint
State Methodc is the minimizer ofthe error functional
∇ · (f (x)∇u(x)) = 0 Function Inverse ProblemNeural Network
Functional Approximatorf (x) ≈ NNw (x)
∇ · (f (u)∇u(x)) = 0 Relation Inverse ProblemResidual Learning
Physics Constrained Learning (PCL)f (u) ≈ NNw (u)
∇ · ($∇u(x)) = 0 Stochastic Inverse ProblemPhysical Generative Neural Networks
(PhysGNN)$ = NNw (vlatent)
ADCME ML for Computational Engineering 38 / 1
A General Approach to Inverse Modeling
ADCME ML for Computational Engineering 39 / 1
Reference
Methodology and Implementation:
Physics Constrained Learning for Data-driven Inverse Modeling fromSparse Observations (Core techniques!)A General Approach to Seismic Inversion with AutomaticDifferentiationTime-lapse Full-waveform Inversion for Subsurface Flow Problems withIntrusive Automatic Differentiation
Consistutive Modeling:Learning Constitutive Relations from Indirect Observations Using DeepNeural NetworksLearning Constitutive Relations using Symmetric Positive DefiniteNeural NetworksInverse Modeling of Viscoelasticity Materials using Physics ConstrainedLearning
Learning Spatially-varying Fields:Solving Inverse Problems in Steady State Navier-Stokes Equationsusing Deep Neural Networks
ADCME ML for Computational Engineering 40 / 1