Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the...

20
Individual Sections of the Book Inverse Problems: Exercices With mathematica, matlab, and scilab solutions Albert Tarantola 1 Université de Paris, Institut de Physique du Globe 4, place Jussieu; 75005 Paris; France E-mail: [email protected] March 12, 2007 1 © A. Tarantola, 2006. Students and professors are invited to freely use this text.

Transcript of Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the...

Page 1: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

Individual Sections of the Book

Inverse Problems: Exercices

With mathematica, matlab, and scilab solutions

Albert Tarantola1

Université de Paris, Institut de Physique du Globe4, place Jussieu; 75005 Paris; France

E-mail: [email protected]

March 12, 2007

1© A. Tarantola, 2006. Students and professors are invited to freely use this text.

Page 2: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

14 General Introduction

1.8 Gaussian Distribution with Exponential Covariance

1.8.1 Generation of Random Gaussian Numbers

There are two common definitions for the error function function:

Φ(x) =1√2π

∫ x

−∞dt e−t2/2 ; Erf x =

2√π

∫ x

0dt e−t2

, (1.31)

that are related viaΦ(x) =

12

(1 + Erf

x√2

). (1.32)

While the first definition is used in probability theory (it is the cumulative probability asso-ciated to the standard Gaussian distribution), other branches of mathematics use the seconddefinition. Φ(x) grows monotonically from 0 to 1 , and Erf x grows monotonically from−1 to +1 . We then have the following

Property 1.1 If y is a random variable in the range (0, 1) with constant probability density, thenthe random variable

x = Φ-1(y) (1.33)

is a Gaussian random variable with zero mean and unit variance. Equivalently, if y is a randomvariable in the range (−1, +1) with constant probability density, then the random variable

x =√

2 Erf-1y (1.34)

is a Gaussian random variable with zero mean and unit variance.

Note: I shall demonstrate this here. Note: I have to warn the reader about the problems withpseudo-random number generators. Note: I shall here refer to figure 1.4.

-4 -2 0 2 40

0.2

0.4

0.6

0.8

1

if constant pdf here

then standardGaussian therey

= !

(x)

x = !-1(y)

cumulative Gaussian function

if constant pdf here

then standardGaussian there

-4 -2 0 2 4-1

-0.5

0

0.5

1

y =

Erf x

/"2_

x = "2 Erf-1 y

_

error function

Figure 1.4: If a generator of random numbers with constant probability density is available,the use of the inverse error function transforms these numbers into numbers having a Gaus-sian probability density.

Let us implement this. The command

Page 3: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

1.8 Gaussian Distribution with Exponential Covariance 15

Do[ u[i] = Sqrt [2] InverseErf[Random[Real ,{ -1 ,+1}]] , {i,1 ,5000} ]

produces 5000 random numbers, whose histogram is displayed in figure 1.5. We clearly seethat the numbers have a Gaussian distribution (the theoretical Gaussian is superimposed onthe histogram).

Figure 1.5: Histogram of the 5000 numbers gen-erated using the inverse error function, with thetheoretical Gaussian superimposed.

-4 -2 0 2 40

200

400

600

800

1000

1.8.2 Generating a Random Vector with Prescribed Mean and Covariance Matrix

Let u be a Gaussian random vector with mean u0 and covariance Cu :

u = u0 ; (u− u0) (u− u0)t = Cu . (1.35)

Defining the random vector

v = L u , (1.36)

it is easy to see6 that v is also a Gaussian vector, whose mean and covariance are

v0 = L u0 ; Cv = L Cu Lt . (1.37)

This implies, in particular, the following —quite practical— property:

Property 1.2 If u is a “Gaussian white noise”, i.e., if u has zero mean, u0 = 0 , and unit covari-ance, Cu = I , then the vector

v = v0 + L u (1.38)

is a Gaussian random vector with mean v0 and covariance Cv = L Lt .

The u here considered is, of course, the u we learn to generate in the previous section.Examples of the random function with prescribed covariance are in section 1.8.6.

6For v = L u = L u = Lu0 , and Cv = (v− v0) (v− v0)t = (L u− L u0) (L u− L u0)t =L (u− u0) (u− u0)t Lt = L (u− u0) (u− u0)t Lt = L Cu Lt .

Page 4: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

16 General Introduction

1.8.3 Distance from a Realization to the Mean Realization

Consider a Gaussian probability density, with mean v0 and covariance matrix Cv . To anyrandom realization v of this Gaussian (without any other constraint) we can associate therandom quantity

χ2 = (v− v0)t C-1v (v− v0) (1.39)

(note: explain here that this is the squared distance between v and v0 ). Random realizationsof the vector v produce random realizations of the real quantity χ2 , so we may ask: whichis the probability distribution of this quantity? It is well-known, and bears a special name:the chi-squared probability density, that is given by (note: I will give a reference here)

f (x; ν) =1

2ν/2 Γ(ν/2)xν/2−1 e−x/2 (1.40)

where, to avoid any confusion, I write the variable x instead of χ2 . The parameter ν isthe “number of degrees of freedom”. In the case considered here, it equals the number ofcomponents of the vector v . If all the components are not random, but k constraints havebeen imposed (statistical mean value constrained, one value imposed, etc.), then, the numberof degrees of freedom is ν = dim(v)− k .

For large values of ν the chi-squared distribution is close to a Gaussian distribution, withmean ν and standard deviation

√2ν . We can easily check this. The command

Do[ chi2[w] = Sum[ (Sqrt [2] InverseErf[Random[Real ,{ -1 ,+1}]])^2, {i,1 ,5000} ] , {w,1 ,2000} ]

makes 2000 experiments, each of which consisting in generating 5000 values of a Gaussianrandom variable with zero mean and unit variance, and computing the distance from thevector so obtained to the vector zero (i.e., making the sum of the squares of the values). Thehistogram of the 200 values so obtained is displayed in 1.6. This is practically a Gaussiandistribution with mean 5000 and standard deviation

√2 5000 = 100 .

Figure 1.6: Histogram of the values of χ2 gen-erated from 2000 experiments, each consistingin generating 5000 Gaussian pseudo-randomvariables. This is practically a Gaussian distri-bution with mean 5000 and standard deviation√

2 5000 = 100 .4700 4800 4900 5000 5100 5200 5300

0

100

200

300

400

Note: I have to explain here that is u is a Gaussian random vector with zero mean andunit covariance, and if we generate v via equation 1.38, then7

χ2 = (v− v0)t C-1v (v− v0) = ut u . (1.41)

7For χ2 = (v− v0)t C-1v (v− v0) = (v− v0)t (L Lt)-1 (v− v0) = ( L-1 (v− v0) )t ( L-1 (v− v0) ) = ut u .

Page 5: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

1.8 Gaussian Distribution with Exponential Covariance 17

1.8.4 The Exponential Covariance Operator and its Square Root

Note: I have to explain here what is the Gaussian distribution. In its elementary version, theexponential function is defined as

c(x, x′) = σ2 e−|x−x′|/L , (1.42)

where σ2 is the variance and L is the correlation length. We shall encounter this as a limit ofa more general covariance function, where the variance starts at zero and tends to σ2 onlyasymptotically:

c(x, x′) = σ2(

1− exp(−2 min(x, x′)L

))

e−|x−x′|/L . (1.43)

The variance at different points is C(0, 0) = 0 , C(L, L) ≈ 0.63 σ2 , C(2L, 2L) ≈ 0.86 σ2 ,and C(3L, 3L) ≈ 0.95 σ2 . We shall find an interpretation of this in a moment. A discretization

x = i ∆x (1.44)

produces the covariance matrix C(i, j) = c(i ∆x, j ∆x) . The introduction of the quantity

a = exp(−∆x/L) (1.45)

allows to express the covariance matrix as

Ci(i, j) = σ2 (1− a2 min(i,j)) a|i−j| . (1.46)

We shall define the square root of C via the Cholesky decomposition

C = L Lt , (1.47)

where Li is lower triangular. It is easy to check that this leads to

(L)ij =

{if i ≤ j (L)ij = w aj−i

if i > j (L)ij = 0 ,(1.48)

whereb = σ

√1− a2 . (1.49)

The knowledge of L shall later be useful for generating pseudo-random realizations of aGaussian distribution with covariance Ci . It is easy to see8 that the computation of a vector

m = L w (1.50)

8The simplest way to see this is to remark that the explicit expression for the matrix representing this operator

is L = b

1 0 0 0 0

. . .a 1 0 0 0

. . .a2 a 1 0 0

. . .a3 a2 a 1 0

. . .a4 a3 a2 a 1

. . .. . .. . .

. . .. . .

. . .. . .

. The inverse matrix is L-1 = 1

b

1 0 0 0 0

. . .-a 1 0 0 0

. . .0 -a 1 0 0

. . .0 0 -a 1 0

. . .0 0 0 -a 1

. . .. . .. . .

. . .. . .

. . .. . .

.

Page 6: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

18 General Introduction

can be performed via the iterative algorithm

m1 = b w1

mi = a mi−1 + b wi ; i = 2, 3, . . . , n .(1.51)

So, in fact, we never need to actually build the matrix L . The inverse relation

w = L-1 m (1.52)

corresponds to the algorithm

w1 = (1/b) m1

wi = (1/b) ( mi − a wi−1 ) ; i = 2, 3, . . . , n(1.53)

(the matrix corresponding to this algorithm is in footnote 8).The transpose operation

w = Lt µ (1.54)

is to be implemented via the algorithm

wn = b µn

wi = a wi+1 + b µi ; i = n− 1, n− 2, . . . , 1 ,(1.55)

while the inverse relationµ = L-t w (1.56)

corresponds to

µn = (1/b) wn

µi = (1/b) ( wi − a wi+1 ) ; i = n− 1, n− 2, . . . , 1 .(1.57)

When needing to compute a vector

m = C µ , (1.58)

one uses the decomposition C = L Lt , that gives m = (L Lt) µ = L (Lt µ) , to, in fact,successively evaluate

w = Lt µ ; m = L w , (1.59)

using, for this, the two algorithms 1.54 and 1.51.Note: decide where to put figures 1.7 and 1.8.

Figure 1.7: Note: caption to be written.

C-1 = L-t L-1

C = L Lt

L-1

L

L-t

Lt

space of“white noises”

model spacedual ofmodel space

Page 7: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

1.8 Gaussian Distribution with Exponential Covariance 19

Figure 1.8: Note: caption to be written.

Ci-1 = Lf-1 Li-1

Ci = Li Lf

Li-1

LiLf-1

Lf

space of“white noises”

model spacedual ofmodel space

Cf-1 = Li-1 Lf-1

Cf = Lf Li

Lf-1

Lf

Li-1

Li

space of“white noises”

model spacedual ofmodel space

1.8.5 Interpretation as a Random Walk

Assume that we want to create a vector m that we want to be a random realization of aGaussian distribution with zero mean and covariance C = L Lt .

We first use the property 1.1 (page 14) to generate a vector w whose components areall (independent) random realizations of a Gaussian one-dimensional distribution with zeromean and unit variance, excepted the first component, at which we give the value zero:

w1 = 0 . (1.60)

Then, we use the property 1.2 (page 1.2) that tells us that the vector m = L w shall bea random realization of a Gaussian distribution with zero mean and covariance C , as wewant. To evaluate m = L w we use the algorithm 1.51, that now gives

m1 = 0mi = a mi−1 + b wi ; i = 2, 3, . . . , n .

(1.61)

The second on these equations can be interpreted as a random walk, where, at each step, thewalker make a random move centered at a times the previous position, the jump having aGaussian distribution having zero mean and standard deviation b . As the quantity a (seeformula 1.45) is typically close to, but smaller than, one, this means that the random walkerhas a tendency to stay around the zero. Setting a = 1 gives a pure (Gaussian) random walk,without any tendency to stay near the value zero.

Questions for Andre Journel: we see that pseudo-random realizations of a one-dimensionalGaussian random field with exponential covariance can be obtained as results of a “Markovchain” (the value at point i depends only on the value at point i − 1 ). Because of this,pseudo-random realizations can be obtained extremely fast. Questions:

• Besides the exponential covariance, which are the other covariance functions can beobtained as the result of a Markov chain?

• What is the value at point i depends only on the values at point i− 1 , i− 2 , . . . i− k ?

Page 8: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

20 General Introduction

• What happens in 2D and 3D?

1.8.6 Random Functions with Exponential Covariance

Executable notebook athttp://www.ipgp.jussieu.fr/~tarantola/exercices/chapter_01/ExponentialCovariance.nb

To illustrate the algorithms above, let us generate a pseudo-random realizations of aGaussian random function (in fact, a discrete function, defined at 5000 points) with zeromean and exponential covariance.

After initializing the pseudo-random generator,

SeedRandom [1]

the commands

imin = 1 ; imax = 5000 ;p[imin] = 0. ;Do[ p[i] = Sqrt [2] InverseErf[Random[Real ,{ -1 ,1}]] , {i,imin+1,imax} ]

produce (for the reasons explained above) the Gaussian white noise displayed in figure 1.9.Then, the commands

(* Values: a=Exp[-1.], a=Exp[-0.1], a=Exp[-0.01], a=Exp [ -0.001] *)s = 5. ; a = Exp[-1.] ; w = Sqrt[1-a^2]/a ;r[imin] = w p[imin];Do[ r[i] = a r[i-1] + w p[i] , {i,imin+1,imax} ]

produce the functions in figures 1.10–1.13, that are the result we wanted to obtain.

0 1000 2000 3000 4000 5000

-4

-2

0

2

4

Figure 1.9: Discrete version (5000 points) of white noise with zero mean and unit variance (infact, 5000 independent random realizations of a Gaussian variable with zero mean and unitvariance). For better comparison with subsequent figures, the first value is of the sequenceis fixed to zero.

If the first value of the initial random sequence (the white noise) is p1 , the first value ofthe random sequence with exponential covariance is r1 = σ

√1− a2 . As we have imposed

p1 = 0 , then r1 = 0 , as can be seen in all the figures 1.10–1.13. As the white noise sequenceis the same for all the figures, all the sequences have a conspicuous similarity.

Note: explain again what follows. There is a random walker that, at each step, decidesrandomly to “go up” or to “go down”. The move has a Gaussian distribution, centered at

Page 9: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

1.8 Gaussian Distribution with Exponential Covariance 21

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

Figure 1.10: Pseudo-random realization of a (discrete) Gaussian random function with expo-nential covariance. The standard deviation is σ = 5 , and the correlation length is L = 1 .With such a small correlation length, the sequence looks like white noise (but it is not). Thegrid has 5000 points, and L0 = ∆x = 1 .

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

Figure 1.11: Same as figure 1.10, but with a correlation length L = 10 . It is clear that valuesat points separated by a few correlations lengths are practically independent.

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

Figure 1.12: Same as figure 1.10, but with a correlation length L = 100 . With a correlationlength as large as in here, any experimental measure of the mean value or of the correla-tion length (assuming stationarity and using two-point statistics) would not give the correctvalues (a longer sequence would be necessary).

Page 10: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

22 General Introduction

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

Figure 1.13: Same as figure 1.10, but with a correlation length L = 1000 . The values are nowstrongly correlated all along the sequence (remember that the intial value has been fixed tozero).

a times the previous position (the constant a is close to, but smaller than one, from wherea tendency, in the long-term, for the random walker to approach the mean value), and withstandard deviation σ

√1− a2 .

1.8.7 Conditional Random Functions with Exponential Covariance

Assume that we have the a priori information that we deal with random realizations of aGaussian distribution with zero mean and exponential covariance (standard deviation 5,correlation length 100). If we now learn the “datum” that the value of the actual function atgrid point 2000 is 5± 1 , we can just. . . (note: explain this).

The code

rnd := Sqrt [2] InverseErf[Random[Real ,{-1,1}]]s = 5. ; a = Exp [ -0.001] ; w = s Sqrt[1-a^2] ;r[imin] = w 0;count = 0;Do[{

Do[ r[i] = a r[i-1] + w rnd , {i,imin+1,imax} ] ,If[ Abs[r[2000] -5] < 1 , PLOT[r] ]}, {200}]

produces the result shown in figure 1.14.If instead of the “hard datum” 5± 1 we model the undertainty using a Laplacian distri-

bution. . . (note: contiue this).

1.8.8 Provisional Example

Executable notebook athttp://www.ipgp.jussieu.fr/~tarantola/exercices/chapter_01/TwoData.nb

Note: what follows has to be placed elsewhere.I examine here a one-dimensional random Gaussian field, with zero mean and exponen-

tial covariance (unit variance). Figure 1.15 displays some random realizations of such a field.We “condition” this random field by using two “uncertain data”: that the function must passthrough the two points indicated in blue (uncertainty bars shown).

Page 11: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

1.8 Gaussian Distribution with Exponential Covariance 23

0 1000 2000 3000 4000 5000

-15

-10

-5

0

5

10

15

Figure 1.14: From 200 trial curves, 20 of them satisfy the condition that the value at grid point2000 equals 5± 1 .

Figure 1.15: Note: caption to be written.

0 20 40 60 80 100

-2

-1

0

1

2

I solve the problem using a standard least-squares formulation, this providing the func-tion shown in red in figure 1.16. This is to be interpreted as the mean of the posterior Gaus-sian, whose covariance is also provided by the lest-squares formulas. Given the mean andthe covariance, we can generate the radom realizations shown in the figure. The same wouldbe obtained using “krigeage” or using the method in Thomas paper (I guess).

Figure 1.16: Note: caption to be written.

0 20 40 60 80 100

-2

-1

0

1

2

0 20 40 60 80 100

-2

-1

0

1

2

The red line at the right in figure 1.17 has been obtained using the same least-squaresformulation, but using an iterative, steepest descent algorithm (initialized at the prior mean).Is the same red line obtained above. When the same iterative algorithm is not intialized at theprior mean, but it is intialized at the individual random realizations of figure 1.15 the linesin black are obtained, this providing, to a good approximation, the same posterior randomrealizations obtained above.

Note: the same random seeds have been used in the three computations. To obtain the

Page 12: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

24 General Introduction

Figure 1.17: Note: caption to be written.

0 20 40 60 80 100

-2

-1

0

1

2

0 20 40 60 80 100

-2

-1

0

1

2

last figure, I have had to manually tune the number of iterations: if their number tends toinfinity, all the lines tend to the red one. I don’t yet underdstands this. It may mean that theproposed conjecture is only an approximation. If this works in Thomas-Geophysics problem,the computational cost may be greatly reduced. Thomas should look at this.

Page 13: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

Exponential covariance

Input package

<< Histograms`;

Using matrices

H* Example for 5 x 5 matrices *Ln = 5;

Initial value given

H* Operator L with initial condition built-in *L

Li = s 1 - a2

1 0 0 0 0

a 1 0 0 0

a2 a 1 0 0

a3 a2 a 1 0

a4 a3 a2 a 1

;

H* The transpose operator *L

Lit = s 1 - a2

1 a a2 a3 a4

0 1 a a2 a3

0 0 1 a a2

0 0 0 1 a

0 0 0 0 1

;

H* The covariance operator Hwith initial conditionL is L L^t *L

covi = s^2

I1 - a2M a I1 - a2M a2 I1 - a2M a3 I1 - a2M a4 I1 - a2M

a I1 - a2M I1 - a4M a I1 - a4M a2 I1 - a4M a3 I1 - a4M

a2 I1 - a2M a I1 - a4M I1 - a6M a I1 - a6M a2 I1 - a6M

a3 I1 - a2M a2 I1 - a4M a I1 - a6M I1 - a8M a I1 - a8M

a4 I1 - a2M a3 I1 - a4M a2 I1 - a6M a I1 - a8M I1 - a10M

;

[email protected] - coviD

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

H* Analytical expression for the covariance function *Lcv@i_, ii_D := s^2 H1 - a^H2 Min@i, iiDLL Exp@Log@aD Abs@i - iiDDTable@Simplify@cv@i, iiD - covi@@i, iiDDD, 8i, 1, n<, 8ii, 1, n<D

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

H* The inverse of the operator L with initial condition built-in *L

iLi =1

s 1 - a2

1 0 0 0 0

-a 1 0 0 0

0 -a 1 0 0

0 0 -a 1 0

0 0 0 -a 1

;

H* Check of the expression *LiLi.Li - IdentityMatrix@nDH* The inverse of the covariance operator with initial condition built-in *L

icovi =1

s2 I1 - a2M

1 + a2 -a 0 0 0

-a 1 + a2 -a 0 0

0 -a 1 + a2 -a 0

0 0 -a 1 + a2 -a

0 0 0 -a 1

;

[email protected]* Check of the expression *LFullSimplify@[email protected] - icoviD

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

881, 0, 0, 0, 0<, 80, 1, 0, 0, 0<, 80, 0, 1, 0, 0<, 80, 0, 0, 1, 0<, 80, 0, 0, 0, 1<<

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

2 ExponentialCovariance.nb

Page 14: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

Final value given

H* Operator L with final condition built-in *L

Lf = s 1 - a2

1 a a2 a3 a4

0 1 a a2 a3

0 0 1 a a2

0 0 0 1 a

0 0 0 0 1

;

H* But the operator L with final condition built-in is just... *LH* ...the transpose of the operator L with initial conditon given *LLf - Transpose@LiDH* The covariance operator with final condition built-in *L

covf = s^2

I1 - a10M a I1 - a8M a2 I1 - a6M a3 I1 - a4M a4 I1 - a2M

a I1 - a8M I1 - a8M a I1 - a6M a2 I1 - a4M a3 I1 - a2M

a2 I1 - a6M a I1 - a6M I1 - a6M a I1 - a4M a2 I1 - a2M

a3 I1 - a4M a2 I1 - a4M a I1 - a4M I1 - a4M a I1 - a2M

a4 I1 - a2M a3 I1 - a2M a2 I1 - a2M a I1 - a2M I1 - a2M

;

[email protected]@LfD - covfD

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

H* The inverse of the operator L with final condition built-in *L

iLf =1

s 1 - a2

1 -a 0 0 0

0 1 -a 0 0

0 0 1 -a 0

0 0 0 1 -a

0 0 0 0 1

;

H* Check of the expression *LiLf.Lf - IdentityMatrix@nDH* The covariance operator with final condition built-in *L

icovf =1

s2 I1 - a2M

1 -a 0 0 0

-a 1 + a2 -a 0 0

0 -a 1 + a2 -a 0

0 0 -a 1 + a2 -a

0 0 0 -a 1 + a2

;

FullSimplify@[email protected] - icovfD

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

880, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<, 80, 0, 0, 0, 0<<

ExponentialCovariance.nb 3

Using algorithms

imin = 1; imax = 5; w = s Sqrt@1 - a^2D;

H* Implementation of Li then its inverse iLi *Lr@iminD = w p@iminD;Do@ r@iD = a r@i - 1D + w p@iD , 8i, imin + 1, imax<DSimplify@r@1DDSimplify@r@2DDSimplify@r@3DDSimplify@r@4DDSimplify@r@5DDpp@iminD = H1ê wL r@iminD;Do@pp@iD = H1ê wL Hr@iD - a r@i - 1DL , 8i, imin + 1, imax<DSimplify@pp@1DDSimplify@pp@2DDSimplify@pp@3DDSimplify@pp@4DDSimplify@pp@5DD

1 - a2 s p@1D

1 - a2 s Ha p@1D + p@2DL

1 - a2 s Ia2 p@1D + a p@2D + p@3DM

1 - a2 s Ia3 p@1D + a2 p@2D + a p@3D + p@4DM

1 - a2 s Ia4 p@1D + a3 p@2D + a2 p@3D + a p@4D + p@5DM

p@1D

p@2D

p@3D

p@4D

p@5D

4 ExponentialCovariance.nb

Page 15: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

H* Implementation of Lf then its inverse iLf *Lr@imaxD = w p@imaxD;Do@r@iD = a r@i + 1D + w p@iD, 8i, imax - 1, imin, -1<DSimplify@r@1DDSimplify@r@2DDSimplify@r@3DDSimplify@r@4DDSimplify@r@5DDpp@imaxD = H1ê wL r@imaxD;Do@pp@iD = H1ê wL Hr@iD - a r@i + 1DL , 8i, imax - 1, imin, -1<DSimplify@pp@1DDSimplify@pp@2DDSimplify@pp@3DDSimplify@pp@4DDSimplify@pp@5DD

1 - a2 s Hp@1D + a Hp@2D + a Hp@3D + a Hp@4D + a p@5DLLLL

1 - a2 s Hp@2D + a Hp@3D + a Hp@4D + a p@5DLLL

1 - a2 s Hp@3D + a Hp@4D + a p@5DLL

1 - a2 s Hp@4D + a p@5DL

1 - a2 s p@5D

p@1D

p@2D

p@3D

p@4D

p@5D

ExponentialCovariance.nb 5

H* Applying Li^t = Lf then Li *Lr@imaxD = w p@imaxD;Do@r@iD = a r@i + 1D + w p@iD, 8i, imax - 1, imin, -1<Dpp@iminD = w r@iminD;Do@ pp@iD = a pp@i - 1D + w r@iD , 8i, imin + 1, imax<DH* Applying the covariance operator *LDo@ ppp@iD = Sum@covi@@i, jDD p@jD, 8j, imin, imax<D, 8i, imin, imax<DSimplify@ppp@1D - pp@1DDSimplify@ppp@2D - pp@2DDSimplify@ppp@3D - pp@3DDSimplify@ppp@4D - pp@4DDSimplify@ppp@5D - pp@5DD

0

0

0

0

0

H* Applying the inverse of Li then the inverse of Lf *Lr@iminD = H1ê wL p@iminD;Do@r@iD = H1ê wL Hp@iD - a p@i - 1DL , 8i, imin + 1, imax<Dpp@imaxD = H1ê wL r@imaxD;Do@pp@iD = H1ê wL Hr@iD - a r@i + 1DL , 8i, imax - 1, imin, -1<DH* Applying the inverse covariance operator *LDo@ ppp@iD = Sum@icovi@@i, jDD p@jD, 8j, imin, imax<D, 8i, imin, imax<DSimplify@ppp@1D - pp@1DDSimplify@ppp@2D - pp@2DDSimplify@ppp@3D - pp@3DDSimplify@ppp@4D - pp@4DDSimplify@ppp@5D - pp@5DD

0

0

0

0

0

6 ExponentialCovariance.nb

Page 16: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

Random Functions: check of the statistics

SeedRandom@1Dimin = 1; imax = 5000;

p@iminD = 0.`;

DoBp@iD = 2 InverseErf@RandomReal@8-1, 1<DD, 8i, imin + 1, imax<F

ListPlotBTable@p@iD, 8i, imin, imax<D, PlotRange Ø 8-4.5`, 4.5`<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-4

-2

0

2

4

chisquare = Sum@p@iD^2, 8i, imin, imax<D

4842.69

h1 = Histogram@Table@p@iD, 8i, imin, imax<D,HistogramCategoriesØ 8-4.5`, -4, -3.5`, -3, -2.5`, -2, -1.5`,

-1, -0.5`, 0, 0.5`, 1.`, 1.5`, 2.`, 2.5`, 3.`, 3.5`, 4.`, 4.5`<Dh2 = PlotB1000 ‰-

rr2

2 , 8rr, -3, 3<FShow@8h1, h2<, PlotRange Ø 88-4.5`, 4.5`<, 8-50, 1050<<,Axes Ø False, Frame Ø True, GridLines Ø 880<, 80<<D

-4 -2 0 2 4

0

200

400

600

800

1000

f@u_, k_D :=1

2kê2 Gamma@kê2D ukê2-1 ‰-uê2

ExponentialCovariance.nb 7

SeedRandom@1DDateList@D

DoBchi@wD = ‚i=imin

imax

J 2 InverseErf@RandomReal@8-1, 1<DDN2

, 8w, 1, 2000<F

DateList@D

82007, 8, 27, 13, 16, 54.407618<

82007, 8, 27, 13, 50, 41.233407<

hh1 = Histogram@Table@chi@wD, 8w, 1, 2000<D,HistogramCategoriesØ 84600, 4650, 4700, 4750, 4800, 4850,

4900, 4950, 5000, 5050, 5100, 5150, 5200, 5250, 5300, 5350, 5400<Dhh2 = Plot@100000 f@u, 5000D, 8u, 4600, 5400<, PlotRange Ø AllD

PlotBf@u, 5000D -1

2 p ss

ExpB-1

2 Hu - 5000L2

ss2F ê. ss -> 2µ5000 , 8u, 4600, 5400<F

Show@8hh1, hh2<, PlotRange Ø 884600, 5400<, 8-20, 440<<,Axes Ø False, Frame Ø True, GridLines Ø 885000<, 80<<D

4700 4800 4900 5000 5100 5200 53000

100

200

300

400

4800 5000 5200 5400

100

200

300

400

8 ExponentialCovariance.nb

Page 17: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

4800 5000 5200 5400

-0.00003

-0.00002

-0.00001

0.00001

0.00002

0.00003

4600 4800 5000 5200 5400

0

100

200

300

400

ExponentialCovariance.nb 9

Random Functions: check of the spaces

SeedRandom@1D Himin = 1; imax = 5000;L Hp@iminD = 0.`;LDoBp@iD = 2 InverseErf@RandomReal@8-1, 1<DD, 8i, imin + 1, imax<F

ListPlotBTable@p@iD, 8i, imin, imax<D, PlotRange Ø 8-4.5`, 4.5`<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

s = 5.`; a = ‰-0.001`; w = s 1 - a2 ; Hr@iminD = w p@iminD;LDo@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-21, 21<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-4

-2

0

2

4

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

10 ExponentialCovariance.nb

Page 18: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

SeedRandom@1D Himin = 1; imax = 5000;L Hp@iminD = 0.`;LDoBp@iD = 2 InverseErf@RandomReal@8-1, 1<DD, 8i, imin + 1, imax<F

ListPlotBTable@p@iD, 8i, imin, imax<D, PlotRange Ø 8-4.5`, 4.5`<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

s = 5.`; a = ‰-0.001`; w = s 1 - a2 ; r@imaxD =p@imaxD

w;

DoBr@iD =p@iD - a p@i + 1D

w, 8i, imax - 1, imin, -1<F

ListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-21, 21<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

Hpp@imaxD = w r@imaxD;L Do@pp@iD = a pp@i + 1D + w r@iD, 8i, imax - 1, imin, -1<DHppp@iminD = w pp@iminD;L Do@ppp@iD = a ppp@i - 1D + w pp@iD, 8i, imin + 1, imax<DListPlotBTable@ppp@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-21, 21<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-4

-2

0

2

4

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

Random Functions: going on

SeedRandom@1Dimin = 1; imax = 5000;

p@iminD = 0.`;

DoBp@iD = 2 InverseErf@RandomReal@8-1, 1<DD, 8i, imin + 1, imax<F

ExponentialCovariance.nb 11

ListPlotBTable@p@iD, 8i, imin, imax<D, PlotRange Ø 8-4.5`, 4.5`<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-4

-2

0

2

4

s = 5.`; a = ‰-0.0001`; w = s 1 - a2 ;

r@iminD = w p@iminD;Do@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

s = 5.`; a = ‰-0.001`; w = s 1 - a2 ;

r@iminD = w p@iminD;Do@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

12 ExponentialCovariance.nb

Page 19: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

s = 5.`; a = ‰-0.01`; w = s 1 - a2 ;

r@iminD = w p@iminD;Do@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

s = 5.`; a = ‰-0.1`; w = s 1 - a2 ;

r@iminD = w p@iminD;Do@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

s = 5.`; a = ‰-1.`; w = s 1 - a2 ;

r@iminD = w p@iminD;Do@r@iD = a r@i - 1D + w p@iD, 8i, imin + 1, imax<DListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F

0 1000 2000 3000 4000 5000

-20

-10

0

10

20

ExponentialCovariance.nb 13

DateList@DSeedRandom@33Drnd := 2 InverseErf@RandomReal@8-1, 1<DD

s = 5.`; a = ‰-0.001`; w = s 1 - a2 ;

r@iminD = w 0;

count = 0;

DoB:Do@r@iD = a r@i - 1D + w rnd, 8i, imin + 1, imax<D,IfBAbs@r@2000D - 5D < 1, :count = count + 1, Print@countD,plot@countD = ListPlotBTable@r@iD, 8i, imin, imax<D, PlotRange Ø 8All, 8-22, 22<<,

Joined Ø True, AspectRatio Ø1

4, Frame Ø True, GridLines Ø 88imin, imax<, 8<<F>F,

Print@io, " , ", r@2000D, " , ", DateList@DD>, 8io, 1, 200<F

82007, 8, 27, 13, 50, 46.762308<

1 , 0.650501 , 82007, 8, 27, 13, 50, 47.873502<

Show@8plot@1D, plot@2D, plot@3D, plot@4D, plot@5D, plot@6D,plot@7D, plot@8D, plot@9D, plot@10D, plot@11D, plot@12D, plot@13D,plot@14D, plot@15D, plot@16D, plot@17D, plot@18D, plot@19D, plot@20D<,PlotRange Ø 8-16, 16<, GridLines Ø 881, 5000<, 80, 4, 6<<, Axes Ø FalseD

0 1000 2000 3000 4000 5000

-15

-10

-5

0

5

10

15

Show@8plot@1D, plot@2D, plot@3D, plot@4D, plot@5D, plot@6D,plot@7D, plot@8D, plot@9D, plot@10D, plot@11D, plot@12D, plot@13D,plot@14D, plot@15D, plot@16D, plot@17D, plot@18D, plot@19D, plot@20D<,PlotRange Ø 88900, 3100<, 85 - 12.5, 5 + 9.5<<, GridLines Ø 882000<, 84, 6<<, Axes Ø FalseD

1000 1500 2000 2500 3000

-5

0

5

10

Show@8plot@1D, plot@2D, plot@3D, plot@4D, plot@5D, plot@6D, plot@7D, plot@8D, plot@9D,plot@10D, plot@11D, plot@12D, plot@13D, plot@14D, plot@15D, plot@16D, plot@17D,plot@18D, plot@19D, plot@20D<, PlotRange Ø 880, 2000<, 85 - 13.5, 5 + 10.5<<,GridLines Ø 882000<, 80, 4, 6<<, Axes Ø FalseD

0 500 1000 1500 2000

-5

0

5

10

15

14 ExponentialCovariance.nb

Page 20: Inverse Problems: Exercicestarantola/Files/Professional/Teaching/Stanford/... · Defining the random vector v = Lu, (1.36) it is easy to see6 that v is also a Gaussian vector, whose

Show@8plot@1D, plot@2D, plot@3D, plot@4D, plot@5D, plot@6D, plot@7D, plot@8D, plot@9D,plot@10D, plot@11D, plot@12D, plot@13D, plot@14D, plot@15D, plot@16D, plot@17D,plot@18D, plot@19D, plot@20D<, PlotRange Ø 882000, 5000<, 85 - 13.5, 5 + 10.5<<,GridLines Ø 882000<, 80, 4, 6<<, Axes Ø FalseD

2000 2500 3000 3500 4000 4500 5000

-5

0

5

10

15

ExponentialCovariance.nb 15