Lecture 15 Nonlinear Problems Newton’s Method
description
Transcript of Lecture 15 Nonlinear Problems Newton’s Method
![Page 1: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/1.jpg)
Lecture 15
Nonlinear Problems
Newton’s Method
![Page 2: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/2.jpg)
SyllabusLecture 01 Describing Inverse ProblemsLecture 02 Probability and Measurement Error, Part 1Lecture 03 Probability and Measurement Error, Part 2 Lecture 04 The L2 Norm and Simple Least SquaresLecture 05 A Priori Information and Weighted Least SquaredLecture 06 Resolution and Generalized InversesLecture 07 Backus-Gilbert Inverse and the Trade Off of Resolution and VarianceLecture 08 The Principle of Maximum LikelihoodLecture 09 Inexact TheoriesLecture 10 Nonuniqueness and Localized AveragesLecture 11 Vector Spaces and Singular Value DecompositionLecture 12 Equality and Inequality ConstraintsLecture 13 L1 , L∞ Norm Problems and Linear ProgrammingLecture 14 Nonlinear Problems: Grid and Monte Carlo Searches Lecture 15 Nonlinear Problems: Newton’s Method Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Lecture 17 Factor AnalysisLecture 18 Varimax Factors, Empircal Orthogonal FunctionsLecture 19 Backus-Gilbert Theory for Continuous Problems; Radon’s ProblemLecture 20 Linear Operators and Their AdjointsLecture 21 Fréchet DerivativesLecture 22 Exemplary Inverse Problems, incl. Filter DesignLecture 23 Exemplary Inverse Problems, incl. Earthquake LocationLecture 24 Exemplary Inverse Problems, incl. Vibrational Problems
![Page 3: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/3.jpg)
Purpose of the Lecture
Introduce Newton’s Method
Generalize it to an Implicit Theory
Introduce the Gradient Method
![Page 4: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/4.jpg)
Part 1
Newton’s Method
![Page 5: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/5.jpg)
grid searchMonte Carlo Method
are completely undirected
alternativetake directions from the
local propertiesof the error function E(m)
![Page 6: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/6.jpg)
Newton’s Method
start with a guess m(p)near m(p) , approximate E(m) as a parabola and
find its minimum
set new guess to this value and iterate
![Page 7: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/7.jpg)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
50
100
150
200
250
300
350
m
E
mnest mn+1estmGM mE(m)
![Page 8: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/8.jpg)
Taylor Series Approximation for E(m)expand E around a point m(p)
![Page 9: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/9.jpg)
differentiate and set result to zero to find minimum
![Page 10: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/10.jpg)
relate b and B to g(m)
linearizeddata kernel
![Page 11: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/11.jpg)
formula for approximate solution
![Page 12: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/12.jpg)
relate b and B to g(m)
very reminiscentof least squares
![Page 13: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/13.jpg)
what do you do if you can’t analytically differentiate g(m) ?
use finite differences to numerically differentiate
g(m)or
E(m)
![Page 14: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/14.jpg)
first derivative
![Page 15: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/15.jpg)
first derivative
vectorΔm [0, ..., 0, 1, 0, ..., 0]T
need to evaluate E(m) M+1 times
![Page 16: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/16.jpg)
second derivative
need to evaluate E(m) about ½M2 times
![Page 17: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/17.jpg)
what can go wrong?
convergence to a local minimum
![Page 18: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/18.jpg)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
50
100
150
200
250
300
350
m
E
mnest mn+1est mGM mE(m)
localminimum global minimum
![Page 19: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/19.jpg)
analytically differentiatesample inverse problem
di(xi) = sin(ω0m1xi) + m1m2
![Page 20: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/20.jpg)
0 0.5 1 1.5 2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
m2
m1
50
100
150
200
0 5 10 15 200
50
100
iteration
E
0 5 10 15 201
1.2
1.4
iteration
m1
0 5 10 15 201
1.5
2
iteration
m2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
2
4
x
d(A)
(B)
(C)
![Page 21: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/21.jpg)
often, the convergence is very rapid
![Page 22: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/22.jpg)
often, the convergence is very rapid
butsometimes the solution converges to a local minimum
andsometimes it even diverges
![Page 23: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/23.jpg)
mg = [1, 1]’;G = zeros(N,M);for k = [1:Niter]
dg = sin( w0*mg(1)*x) + mg(1)*mg(2); dd = dobs-dg; Eg=dd'*dd; G = zeros(N,2); G(:,1) = w0*x.*cos(w0*mg(1)*x) + mg(2); G(:,2) = mg(2)*ones(N,1); % least squares solution dm = (G'*G)\(G'*dd); % update mg = mg+dm; end
![Page 24: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/24.jpg)
Part 2
Newton’s Method for anImplicit Theory
![Page 25: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/25.jpg)
Implicit Theoryf(d,m)=0with Gaussian
prediction errorand
a priori information about m
![Page 26: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/26.jpg)
to simplify algebragroup d, m into a vector x
![Page 27: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/27.jpg)
0 1 2 3 4 5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
x2
x1
parameter, x2
<x1>
<x2>para
meter, x 1
![Page 28: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/28.jpg)
represent data and a priori model parameters as a Gaussian p(x)
f(x)=0 defines a surface in the space of x
maximize p(x) on this surface
maximum likelihood point is xest
![Page 29: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/29.jpg)
0 1 2 3 4 5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
x2
x1
01
23
45
0
0.1
0.2
0.3
s
P(s)x2x2est
x1est
x1
p(x1)
x1 ML
x 1
<x1 >
f(x)=0
(B)(A)
![Page 30: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/30.jpg)
can get local maxima if f(x) isvery non-linear
![Page 31: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/31.jpg)
0 1 2 3 4 5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
x2
x1
01
23
45
0
0.1
0.2
0.3
s
P(s)x2x2est
x1est
x1
p(x1)x1 ML
x 1
<x1 >
f(x)=0
(B)(A)
![Page 32: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/32.jpg)
mathematical statement of the problem
its solution (using Lagrange Multipliers)
with Fij = ∂fi/∂xj
![Page 33: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/33.jpg)
mathematical statement of the problem
its solution (using Lagrange Multipliers)
reminiscent ofminimum length solution
![Page 34: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/34.jpg)
mathematical statement of the problem
its solution (using Lagrange Multipliers)
oops!x appears in 3 places
![Page 35: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/35.jpg)
solutioniterate !
new value for xis x(p+1)
old value for xis x(p)
![Page 36: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/36.jpg)
special case of an explicit theoryf(x) = d-g(m)
equivalent to solving
using simple least squares
![Page 37: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/37.jpg)
special case of an explicit theory f(x) = d-g(m)
weighted least squares generalized inversewith a linearized data kernel
![Page 38: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/38.jpg)
special case of an explicit theory f(x) = d-g(m)Newton’s Method, but making E+L small
not just E small
![Page 39: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/39.jpg)
Part 3
The Gradient Method
![Page 40: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/40.jpg)
What if you can computeE(m) and ∂E/∂mp
but you can’t compute∂g/∂mp or ∂2E/∂mp∂ mq
![Page 41: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/41.jpg)
mnest mGM
E(m)
![Page 42: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/42.jpg)
mnest mGM
E(m)
you know the directiontowards the minimumbut not how far away it is
![Page 43: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/43.jpg)
unit vector pointing towards the minimum
so improved solution would be
if we knew how big to make α
![Page 44: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/44.jpg)
Armijo’s ruleprovides an acceptance criterion for α
with c≈10-4
simple strategystart with a largish α
divide it by 2 whenever it fails Armijo’s Rule
![Page 45: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/45.jpg)
(A)
(B)(C)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
2
4
dx
m2
m1
0 0.5 1 1.5 2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
220
40
60
80
100
120
140
160
180
200
220
m2
m1
0 50 100 150 2000
50
100
0 50 100 150 2001
1.2
1.4
0 50 100 150 2001
1.5
2
iteration
iteration
iteration
Error,
Em 1
m 2
![Page 46: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/46.jpg)
% error and its gradient at the trial solutionmgo=[1,1]';ygo = sin( w0*mgo(1)*x) + mgo(1)*mgo(2);Ego = (ygo-y)'*(ygo-y);dydmo = zeros(N,2);dydmo(:,1) = w0*x.*cos(w0*mgo(1)*x) + mgo(2);dydmo(:,2) = mgo(2)*ones(N,1);dEdmo = 2*dydmo'*(ygo-y);
alpha = 0.05;c1 = 0.0001;tau = 0.5;
Niter=500;for k = [1:Niter] v = -dEdmo / sqrt(dEdmo'*dEdmo);
![Page 47: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/47.jpg)
% backstep for kk=[1:10] mg = mgo+alpha*v; yg = sin(w0*mg(1)*x)+mg(1)*mg(2); Eg = (yg-y)'*(yg-y); dydm = zeros(N,2); dydm(:,1) = w0*x.*cos(w0*mg(1)*x)+ mg(2);
dydm(:,2) = mg(2)*ones(N,1); dEdm = 2*dydm'*(yg-y); if( (Eg<=(Ego + c1*alpha*v'*dEdmo)) ) break; end alpha = tau*alpha; end
![Page 48: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/48.jpg)
% change in solution Dmg = sqrt( (mg-mgo)'*(mg-mgo) ); % update mgo=mg; ygo = yg; Ego = Eg; dydmo = dydm; dEdmo = dEdm; if( Dmg < 1.0e-6 ) break; endend
![Page 49: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/49.jpg)
often, the convergence is reasonably rapid
![Page 50: Lecture 15 Nonlinear Problems Newton’s Method](https://reader030.fdocuments.us/reader030/viewer/2022013115/568167fc550346895ddd7757/html5/thumbnails/50.jpg)
often, the convergence is reasonably rapid
exceptionwhen the minimum is in along a long shallow valley