Numerical Programming 2 (CSE)

Massimo Fornasier

Fakultat fur MathematikTechnische Universitat Munchen

massimo.fornasier@ma.tum.dehttp://www-m15.ma.tum.de/

Johann Radon Institute (RICAM)Osterreichische Akademie der Wissenschaften

massimo.fornasier@oeaw.ac.athttp://hdsparse.ricam.oeaw.ac.at/

Department of MathematicsTechnische Universitat Munchen

Lecture 9

Why are you here?In this course we shall provides the rudiments (the basics!) of thenumerical solution of (partial) differential equations, i.e, equationsinvolving the derivatives of a (possibly) multivariate functionu : Ω ⊂ Rd → R:

F (x1, . . . , xd , u(x),∂

∂x1u(x), . . . ,

∂

∂xdu(x),

∂2

∂x1∂x1,

∂2

∂x1∂x2, . . . ,

∂2

∂x1∂xd, . . . ) = 0.

Often one of the variables indicates the time, e.g., x1 = t, and theequation governs a phenomena which evolves in time:

∂u

∂t+ F (x , u,Du,D2u, . . . ) = 0.

Differential equations were for thefirst time formulated in the 17th

century by I. Newton (1671) and byG.W. Leibniz (1676).

Just old boring stuff?

Yes, of course!?But also NO, because the history did not stop ...

used by Euler, Maxwell, Boltzmann, Navier, Stokes, Einstein,Prandtl, Schrodinger, Pauli, Dirac,Turing, Black&Scholes ...........

I provide an important modeling tool for the physical sciences,theoretical chemistry, biology, socio-economic sciences,engineering sciences ....

I accessible to numerical simulations

For instance ...Fourier in Theorie analytique de la chaleur (1822) formulated and solvedanalytically the equation governing the heat conduction in homogenous media:

∂u

∂t(t, x)− α

„∂2u

∂x21

(t, x) +∂2u

∂x22

(t, x) +∂2u

∂x23

(t, x)

«= 0.

A similar equation is studied by Fornasier and March in 2007 to recolor ancientItalian frescoes from fragments:

See Restoration of color images by vector valued BV functions and variationalcalculus (M. Fornasier and R. March), SIAM J. Appl. Math., Vol. 68 No. 2,2007, pp. 437-460.

http://www.ricam.oeaw.ac.at/people/page/fornasier/SIAM_Fornasier_

March_067187.pdf

Illustrations of differential equations and some of their(modern) applications

Reference: P. A. Markowich, Applied Partial Differential Equations - A Visual

Approach, Springer, 2006

Slides: http:

//homepage.univie.ac.at/peter.markowich/galleries/vortrag.pdf

I Gas dynamics Boltzmann equationI Fluid/gas dynamics: Navier-Stokes/Euler EquationsI Kinetic modeling of granular flowsI Chemotaxis and formation of biological patternsI Semiconductor modelingI Free boundary problems and interfacesI Reaction-diffusion equationsI Monge-Kantorovich optimal transportationI Wave equationsI Digital image processingI Socio-Economic modeling

From partial differential equations to ordinary differentialequations

As we shall see in the course of this lecture, several partialdifferential equations (PDE) can be reduced, after discretization ofthe space variable, to a system of ordinary differential equations(ODE):

∂u

∂t+ F (t, x , u,Du,D2u, . . . ) = 0⇒ u′h(t) + Fh(u(t)) = 0,

where h is a space discretization parameter.

Hence, the first part of this course is dedicated to the numericalsolution of ODE, while the second part of the course is dedicatedto the solution of some relevant equations of elliptic (e.g., materialelasticity), parabolic (e.g., heat conduction), hyperbolic (e.g., waveevolution) types.

Program of the course in a nutshellODE:

I forward Euler method

I theta method

I Adams methods and more general multi-step methods

I Runge-Kutta methods

I stability and stiffness

I Backward differentiation formulae

I Predictor-corrector methods

PDE:

I finite difference

I numerical solution of the Poisson equation in one dimension

I Poisson equation in two dimensions: five-point method

I finite element method

I spectral method

I finite difference for parabolic equations: the heat equation

I stability issues and analysis by the Fourier method

I finite difference for the advection equation

I Lax-Wendroff method for hyperbolic equations

I hyperbolic systems

References for this course

Besides the slides provided after the lecture online ...Books:

I09 A. Iserles, A First Course in the Numerical Analysis of DifferentialEquations (2nd ed.), Cambridge University Press, 2009.

QSG10 A. Quarteroni, F. Saleri, P. Gervasio, Scientific Computing with Matlaband Octave (3rd ed.), Springer, 2010.

Lecture notes:

F13 M. Fornasier, Numerik der gewohnlichen Differentialgleichungen,http://www-m3.ma.tum.de/foswiki/pub/M3/NumerikDG13/WebHome/

Numerik_2013-08-04.pdf (password required)

QSG10 C. Lasser, Numerical Programming 2 (CSE), synthesis of class lectures,http://www-m3.ma.tum.de/foswiki/pub/M3/Allgemeines/

NumericalCSE12/CSE_SS12.pdf

All you find at the webpage of the course:

http://www-m15.ma.tum.de/Allgemeines/NumericalProgramming

References do not mean ...

that you do not come to the lecture and think to be smart ...

’cause I will NOT let you escape ...

At the end there will be the exam ... and I’ll be waiting for youthere (REMEMBER IT!) ...you better show up! (just a friendly - Italian - invitation not toskip the lectures :-) !)

Let’s start ... why ODE?

As just mentioned, ODE comes often after the space discretizationof PDE.

But ODE have their own dignity and relevance, since, actually,PDE can be derived as the “limit” of sytems of ODE governing theevolution of particles driven by Newton laws:

F = m · a.

Hence, ODE ⇒ PDE ⇒ ODE.

Particle systemsBesides in physics, large particle systems arise in many modernapplications:

Image halftoning via variational

dithering.

Dynamical data analysis: R. palustris

protein-protein interaction network.

Large Facebook “friendship” network

Computational chemistry: molecule

simulation.

Social dynamics

We consider large particle systems ofform:

xi = vi ,

vi =∑N

j=1 H(xj − xi , vj − vi ),

Several“social forces” encoded in theinteraction kernel H:

I Repulsion-attraction

I Alignment

I ...

Possible noise/uncertainty by addingstochastic terms.

Patterns related to different balance of

social forces.

Understanding how superposition of re-iterated binary “socialforces” yields global self-organization.

An example inspired by nature

Mills in nature and in our simulations.

J. A. Carrillo, M. Fornasier, G. Toscani, and F. Vecil, Particle, kinetic,

hydrodynamic models of swarming, within the book “Mathematical modeling

of collective behavior in socio-economic and life-sciences”, Birkhauser (Eds.

Lorenzo Pareschi, Giovanni Naldi, and Giuseppe Toscani), 2010.

http://www.ricam.oeaw.ac.at/people/page/fornasier/bookfinal.pdf

The genearal ODEOur goal is to approximate the solution of the problem

y ′ = f (t, y), t ≥ t0, y(t0) = y0. (1)

(In the previous modeling, yi = (xi , vi ) andf (t, y)i = (vi ,

∑Nj=1 H(xj − xi , vj − vi )).)

I f is a map of [t0,∞)× Rd to Rd and the initial conditiony0 ∈ Rd is a given vector

I f is “nice”, obeying, in a given vector norm, the Lipschitzcondition

‖f (t, x)− f (t, y)‖ ≤ λ‖x − y‖, ∀x , y ∈ Rd , t ≥ t0 (2)

Here λ > 0 is a real constant that is independent of thechoice of x and y . In this case we say that f is a Lipschitzcontinuous function.

I Subject to (2), it is possible to prove that the ODE system(1) possesses a unique solution (see, for instance [Theorem3.5, F13] pag. 52).

Just an idea of the proof of existence and uniqueness(Picard 1890, Lindelof 1894)

Actually (1) can be rewritten by integration

y(t) = y0 +

∫ t

t0

f (s, y(s))ds = F (y(t)). (3)

Hence, a solution y(t) is a fixed point trajectory of the equationy = F (y). How can one solve fixed point equations? Well, if Fwere a contraction, i.e., F is a Lipschitz continuous function withLipschitz constant 0 < Λ < 1, then the iteration

yn+1 = F (yn), n ≥ 0 (4)

converges always to the unique fixed point! Indeed, assume thatsuch fixed point y exists then

‖y−yn+1‖∗ = ‖F (y)−F (yn)‖∗ ≤ Λ‖y−yn‖∗ ≤ Λn‖y−y 0‖∗ → 0, n→∞,

because Λ < 1. The tricky part of the proof is to show that thereexists a fixed point and that there exists always a norm ‖ · ‖∗ whichmakes F a contraction as soon as f is Lipschitz continuous.

Did you noticed that ...... what we wrote in (4) is actually an ALGORITHM?!

OMG! Really, an ALGORITHM?!

... is it a bad thing?!Actually, no, that’s what you are here for! Or, perhaps yes ... Ohwell, nevertheless, an ALGORITHM!

For the brave student: if you find out a way to discretize (4) and a way of

properly “scaling” the equation so that the iteration always converges on a

finite set of time point t0 < t1 < · < tn, then let me know! Actually this course

is (IMPLICITLY) all about this!

From Picard-Lindelof back to EulerInstead of solving globally the fixed point equation (3) by amultitude of iterations (4), we may consider the simpler idea ofsolving it locally, step by step, by iterating the approximation

y(t) = y0+

∫ t

t0

f (s, y(s))ds ≈ y(t0)+(t−t0)f (t0, y(t0)), for t ≈ t0.

(5)Given a sequence t0, t1 = t0 + h, t2 = t0 + 2h, ... , where h > 0 isa time step, we denote by yn a numerical estimate of the exactsolution y(tn), n = 0, 1, . . . . Motivated by (5), we choose

y1 = y0 + hf (t0, y0).

If h is small, it should not be that wrong! But then, why not tocontinue, assuming that we did not that bad before, at t2, t3 andso on. In general, we obtain the recursive scheme

yn+1 = yn + hf (tn, yn), (6)

the celebrated Euler method.

Graphic interpretationConsider the Euler method applied to the logistic equation

y ′ = y(1− y), y(0) =1

10,

with step h = 1:

It’s clear that at each step we produce an error, but our goal is not to avoid

any (numerical error)! Our final goal is to have a practical method that

approximates the analytic solution with increasing accuracy (i.e., decreasing

error) the more computational effort we do.

Convergence of a numerical method

Assume that h > 0 is variable and h→ 0. On each grid t0,t1 = t0 + h, t2 = t0 + 2h, ... we associate a different numericalsequence yn = yn,h, n = 0, 1, . . . , bt∗/hc (not necessarily producedby the Euler method!).

A method is said to be convergent if, for every ODE (1) with aLipschitz function f and every t∗ > 0 it is true that

limh→0

maxn=0,1,...,bt∗/hc

‖yn,h − y(tn)‖ = 0,

where bαc ∈ Z is the integer part of α ∈ R.

Convergence means that, for every Lipschitz function,the numerical solution tends to the true solution as thegrid becomes increasingly fine.

Convergence of the Euler method

TheoremThe Euler method (6) is convergent

Proof. (For this proof we assume that the Taylor expansion of f has uniformlybounded coefficients, i.e., f is analytic, implying that y is analytic as well.) Letus consider the error en,h = yn − y(tn). We shall prove limh→0 ‖en,h‖ = 0.Taylor expansion of y(t)

y(tn+1) = y(tn) + hy ′(tn) +O(h2) = y(tn) + hf (tn, y(tn)) +O(h2).

Subtracting this equation to (6), we obtain

en+1,h = en,h + h[f (tn, yn)− f (tn, y(tn))] +O(h2).

Triangle inequality and (2) imply

‖en+1,h‖ ≤ ‖en,h‖+ h‖f (tn, yn)− f (tn, y(tn))‖+ ch2

≤ (1 + hλ)‖en,h‖+ ch2.

By induction over this estimate we get

‖en,h‖ ≤c

λh[(1 + hλ)n − 1], n = 0, 1, . . .

Convergence of the Euler method continues ...

We notice now that 1 + ξ ≤ eξ for all ξ > 0, hence (1 + hλ) ≤ ehλ and(1 + hλ)n ≤ ehnλ. But n = 0, 1, . . . , bt∗/hc and n ≤ t∗/h, implying

(1 + hλ)n ≤ et∗λ

and‖en,h‖ ≤

h c

λ(et∗λ − 1)

i· h. (7)

Sinceh

cλ

(et∗λ − 1)i

is independent of n and h, it follows

limh→0

max0≤nh≤t∗

‖en,h‖ = 0.

The error bound in (7) tells us that actually Euler’s methodconverges with order q = 1 since it decays as O(hq). However,letus stress that the constant

[cλ(et∗λ − 1)

]is by far over-pessimistic

and should not be used for numerical puroposes (it’s justtheoretical!).

Example

Let’s consider

y ′ = −100y , y(0) = 1, t∗ = 1.

The error estimate (7) would give us the following bound on theerror

|en,h| ≤ 2.69× 1045h,

while the exact and numerical solutions are respecivelyy(t) = e−100t and yn = (1− 100h)n, giving

|yn − yn,h| = |(1− 100h)n − e−100nh|,

which is by far smaller than 2.69× 1045h!

Consistency error and orderEuler’s method can be rewritten

yn+1 − yn − hf (tn, yn) = 0.

How well this equation represents the original solution? Let’s plug the originalsolution in this equation

y(tn+1)− y(tn)− hf (tn, y(tn)) = y(tn+1)− y(tn)− hy ′(tn) = O(h2), (8)

by Taylor’s expansion error. This residual error O(h2) is called the consistencyerror. In general, given an arbitrary time-stepping method

yn+1 = Yn(f , h, y0, y1, . . . , yn),

for the ODE (1), we say that it is of consistency order p if

y(tn+1)− Yn(f , h, y(t0), y(t1), . . . , y(tn)) = O(hp+1).

for every analytic f and n = 0, 1, . . . . Hence, by (8) Euler’s method hasconsistency order 1!

Never confuse the order of convergence with the consistency order! They may

well be different! A method can have a positive consistency order but not be

convergent! (see later)

How to implement Euler’s method in MATLAB

Try out Euler’s method on the logistic equation:

f = @(t,x) x.*(1-x); [t,yy]=ode45(f,[0:5],1/10); y(1)=1/10;h=1; for n=1:5, y(n+1)=y(n)+h*y(n)*(1-y(n)); end,plot([0:5],y,’r*-’,[0:5],yy,’b’) h=1/2; for n=1:10,y(n+1)=y(n)+h*y(n)*(1-y(n)), end, hold on,plot([0:0.5:5],y,’g-*’)

How does Euler’s method discretize a PDE?

Given∂u

∂t− F (t, x , u,Du,D2u, . . . ) = 0

one considers

un+1i − un

i

∆t= Fi (tn, x , u,Du,D2u, . . . ).

Trapezoidal rule alias Crank-Nicolson method

Euler’s method approximates the derivative by a constant in[tn, tn+1], namely by its value at tn. Clearly, the “leftist”approximation is not very good (it’s NOT a political comment!Left can be good :-) )and it makes more sense to make the constant approximation ofthe derivative equal to the average of its values at the endpoints:

y(t) = y0+

∫ t

t0

f (s, y(s))ds ≈ y(t0)+1

2(t−t0)[f (t0, y(t0))+f (t, y(t))],

(9)for t ≈ t0. This brings us to consider the trapezoidal rule or theCrank-Nicolson method

yn+1 = yn +h

2[f (tn, yn) + f (tn+1, yn+1)]. (10)

Consistency order of the Crank-Nicolson method

To obtain the order of (10), we substitute the exact solution,

y(tn+1)− y(tn)− h

2[f (tn, y(tn)) + f (tn+1, y(tn+1))]

= [y(tn) + hy ′(tn) +h2

2y ′′(tn) +O(h3)]

− (y(tn) +h

2y ′(tn) + [y ′(tn) + hy ′′(tn) +O(h2)]) = O(h3).

Therefore the trapezoidal rule is of order 2. This is very good, butwe should be careful (as we shall see later) already to concludethat the method is convergent. Actually we need to prove it.

Convergence of the Crank-Nicolson method

TheoremThe Crank-Nicolson method (10) is convergent.

Proof. (For this proof we assume again that f is analytic.) Let us consider theerror en,h = yn − y(tn). We shall prove limh→0 ‖en,h‖ = 0. Substracting

y(tn+1) = y(tn) +h

2[f (tn, y(tn)) + f (tn+1, y(tn+1))] +O(h3).

(see for instance [Section 2.2.2, F13] pag. 21 for this asymptotic error) to (10),we obtain

‖en+1,h‖ ≤ ‖en,h‖+h

2[‖f (tn, yn)− f (tn, y(tn))‖+ ‖f (tn+1, yn+1)− f (tn+, y(tn+1))‖]

+ ch3

≤ ‖en,h‖+hλ

2[‖en,h‖+ ‖en+1,h‖] + ch3.

By assumption that h > 0 gets so small that hλ < 2, we get

‖en+1,h‖ ≤

1 + 1

2hλ

1− 12hλ

!‖en,h‖+

c

1− 12hλ

!h3.

Convergence of the Crank-Nicolson method continues ...

By induction over this estimate we get

‖en,h‖ ≤c

λh2

" 1 + 1

2hλ

1− 12hλ

!n

− 1

#, n = 0, 1, . . .

Since 0 < hλ < 2, it is true that 1 + 1

2hλ

1− 12hλ

!= 1 +

hλ

1− 12hλ

!≤∞X`=0

1

`!

hλ

1− 12hλ

!`= exp

hλ

1− 12hλ

!.

But n = 0, 1, . . . , bt∗/hc and n ≤ t∗/h, implying

‖en,h‖ ≤

"c

λexp

t∗λ

1− 12hλ

!#· h2. (11)

It followslimh→0

max0≤nh≤t∗

‖en,h‖ = 0.

The error bound in (11) tells us that actually the methodconverges with order q = 2 since it decays as O(hq).

But not so easy ...Notice that the Crank-Nicolson method

yn+1 = yn +h

2[f (tn, yn) + f (tn+1, yn+1)]︸ ︷︷ ︸

:=Fn(yn+1)

.

does not allow for the explicit/direct computation of yn+1 from yn

as it occurs for (6). Indeed it is a so-called implicit method and itrequires the solution of the nonlinear equation (10) in the knownyn+1. How can we do it? Well, for instance, we observe thatactaully yn+1 = Fn(yn+1) is a fixed point of an equation and thatFn is a contraction for h sufficiently small! Therefore we cancompute

yn+1 = limm→∞

ymn+1,

where the sequence (ymn+1)m is generated (similarly to the

Picard-Lindelof iteration!!) by

ym+1n+1 = Fn(ym

n+1), m ≥ 0.

Newton ... again

The solution of the equation

0 = G (yn+1) = yn+1 − Fn(yn+1)

can also be solved by Newton’ method:

ym+1n+1 = ym

n+1 −∇G (ymn+1)−1 · G (ym

n+1), m = 0, 1, . . .

where ∇G (ymn+1) is the Jacobian matrix of G computed at ym

n+1.Again we can compute

yn+1 = limm→∞

ymn+1,

How to implement the Crank-Nicolson method inMATLAB?

Try this out:

f = @(t,x) -x + 2*exp(-t)*cos(2*t); t(1)=0; x(1)=0;y(1)=0; h=1/2; for n=1:(10/h), y(n+1)=fsolve(@(x)x-y(n)-h/2*(f(t(n),y(n))+f(t(n+1),x)),y(n)); end,semilogy(t,abs(y-xx),’k’,t, h2*ones(size(t)),’r:’)

How does the Crank-Nicolson method discretize a PDE?

Given∂u

∂t− F (t, x , u,Du,D2u, . . . ) = 0

one considers

un+1i − un

i

∆t=

1

2[Fi (tn, x , u,Du,D2u, . . . )+Fi (tn+1, x , u,Du,D2u, . . . )]

Example

Euler’s method and the trapezoidal rule, as applied to y ′ = −y + 2e−tcos2t,

y(0) = 0. The logarithm of the error, ln(|yn − y(tn)|), is displayed for h = 1/2

(solid line), h = 10 (broken line) and h = 50 (broken-and-dotted line).

Logarithm of the error

If ‖e‖ ≈ chp, then ln e ≈ ln c + p ln h. Denoting by e(1) and e(2)

the errors corresponding to step sizes h(1) and h(2) respectively, itfollows that

ln e(2) ≈ ln e(1) + p ln(h(2)/h(1)).

The ratio of consecutive step sizes being five, we expect the errorto decay by (at least) a constant multiple of ln 5 ≈ 1.6094 and2 ln 5 ≈ 3.2189 for Euler and the trapezoidal rule respectively.

Midpoint rule

To derive the the average

y ′(t) ≈ 1

2[f (tn, yn) + f (tn+1, yn+1)]

is a sensible choice. Similar reasoning leads, however, to analternative approximation,

y ′(t) ≈ f (tn +1

2h,

1

2(yn + yn+1)), t ∈ [tn, tn+1]

and to the implicit midpoint rule

yn+1 = yn + hf (tn +1

2h,

1

2(yn + yn+1))

Exercise: prove that midpoint rule is of second order and that itconverges! The implicit rule is a special case of the Runge-Kuttamethod (more to come).

The theta-method

Both Euler’s method and the trapezoidal rule fit the generalpattern

yn+1 = yn + h[θf (tn, yn) + (1− θ)f (tn+1, yn+1)], (12)

for θ = 1 and θ = 1/2 respectively.

Consistency order

Substituting the exact solution y(t) into the formula

y(tn+1)− y(tn)− h[θf (tn, y(tn)) + (1− θ)f (tn+1, y(tn+1))]

= y(tn+1)− y(tn)− h[θy ′(tn) + (1− θ)y ′(tn+1)]

= [y(tn) + hy ′(tn) +1

2h2y ′′(tn) +

1

6h3y ′′′(tn)]

− y(tn)− hθy ′(tn) + (1− θ)[y ′(tn) + hy ′′(tn) +1

2h2y ′′′(tn)]

+O(h4)

= (θ − 1

2)h2y ′′(tn) + (

1

2θ − 1

3)h3y ′′′(tn) +O(h4).

Therefore the method is of order 2 for θ = 1/2 (the trapezoidalrule) and otherwise of order one.

Error estimates and convergence

Subtracting (as usual!) the last expression from

yn+1 − yn + h[θf (tn, yn) + (1− θ)f (tn+1, yn+1)] = 0,

one obtains

en+1 = en + θh[f (tn, y(tn) + en)− f (tn, y(tn))]

+ (1− θ)h[f (tn+1, y(tn+1) + en+1)f (tn+1, y(tn+1))]− 1

12 h3y ′′′(tn) +O(h4), θ = 12 ,

+(θ − 12 )h2y ′′(tn) +O(h3), θ 6= 1

2 .

By using the Lipschitz continuity of f and proceeding as alreadydone so far, one can show that the θ-methods are alwaysconvergent, for all θ ∈ [0, 1]. Exercise!

An important byproduct: implicit Euler’s method

The choice θ = 0 is of great practical relevance. The first-orderimplicit method

yn+1 = yn + hf (tn+1, yn+1), n = 0, 1, . . . (13)

is called the backward Euler’s method (or implicit Euler’s method)and is a favourite algorithm for the solution of stiff ODEs (more tocome).

Adams method: introduction

Let us suppose again that yn is the numerical solution attn = t0 + nh. Assume that

ym = y(tm) + O(hs+1), m = 0, 1, . . . , n + s − 1, (14)

where s ≥ 1 is a given integer. Let us specify now how can weadvance the solution from tn−s+1 to tn+s . First write

y(tn+s) = y(tn+s−1)+

∫ tn+s

tn+s−1

y ′(τ)dτ = y(tn+s−1)+

∫ tn+s

tn+s−1

f (τ, y(τ))dτ .

We shall compute the integral by extrapolation, constructing theinterpolation polynomial of f on the nodes f (t`, y(t`)) for` = n, n + 1, n + s − 1 and then compute the integral of thispolynomial between tn+s−1 and tn+s .

Interpolation polynomial

Explicitly,

p(t) =s−1∑m=0

pm(t)f (tn+m, yn+m),

where

pm(t) =s−1∏

`=0,` 6=m

t − tn+`

tn+m − tn+`=

(−1)s−1−m

m!(s − 1−m)!

s−1∏`=0,` 6=m

(t − tn

h− `)

=(−1)s−1−m

m!(s − 1−m)!

s−1∏`=0,`6=m

(t − t0 − hn

h− `)

and the Lagrange interpolation polynomials (see [Section 1.1, F13]pag. 8).

Construction of the scheme

Exercise: show that (14) implies

p(tm) = f (tm, ym) = y ′(tm)+O(hs+1), m = n, n+1, . . . , n+s−1.

(use the Lipschitz continuity of f ). Hence, p interpolates y ′ up toan error of order at least O(hs). By using the theory of [Theorem1.2, F13] we can deduce that

p(t) = y ′(t) +O(hs) = f (t, y(t)) +O(hs), ∀t ∈ [n + s−1, n + s].

Hence if we substitute p(t) to f (t, y(t)) in∫ tn+s

tn+s−1f (τ, y(τ))dτ

and yn+s−1 to y(tn+s−1), we produce and error of order O(hs+1)and we obtain the scheme

yn+s = yn+s−1 + hn+s−1∑m=n

bmf (tn+m, yn+m), (15)

which is, by construction, of consistency order p = s.

On the coefficients of the s-step Adams-Bashforth method

Notice that

bm = h−1

∫ tn+s

tn+s−1

pm(τ)dτ =(−1)s−1−m

m!(s − 1−m)!

∫ 1

0

s−1∏`=0,`6=m

[s−(`+1)−ξ]dξ

do not depend on n or h and can be used for subsequent iterations.

Examples

For s = 1 this scheme coincides with the Euler’s method, whereass = 2 gives explicitly

yn+2 = yn+1 + h[3

2f (tn+1, yn+1)− 1

2f (tn, yn)],

and s = 3 gives

yn+3 = yn+2 + h[23

12f (tn+2, yn+2)− 4

3f (tn+1, yn+1) +

5

12f (tn, yn)],

ComparisonLogarithm of the errors of Euler, 2-step, and 3-step Adams-Basgforth methodsfor the equation

y ′ = y 2, y(0) = 1.

When h is halved, Euler’s error decreases linearly, the error of 2-step AB decays

quadratically and 3-step AB displays cubic decay.

Order of convergence of multistep methods

Write a general s-step method in the form

s∑m=0

amyn+m = hs∑

m=0

bmf (tn+m, yn+m), n = 0, 1, . . . (16)

where am, bm, m = 0, 1, . . . , s, are given constants, independent ofh, n and the underlying equation. It is conventional to normalizeas = 1. When bs = 0 the method is explicit (as in theAdams-Bashford methods), otherwise again implicit. Adapting thedefition of consistency order to these methods we say that (16) hasorder p if

ψ(t, y) =s∑

m=0

amy(t+mh)−hs∑

m=0

bmf (t+mh, y(t+mh)) = O(hp+1),

for h→ 0, for all sufficiently smooth y and there is one functionwhich do not allow to improve this order.

Characterization of the order

The method (16) can be characterized in terms of the polynomials

ρ(w) :=s∑

m=0

amwm, σ(w) :=m∑

s=0

bmwm.

TheoremThe multistep method is of order p ≥ 1 if and only if there existsc 6= 0 such that

ρ(w)− σ(w) ln w = c(w − 1)p+1 +O(|w − 1|p+2), w → 1.

ProofWe assume that y is analytic and that its radius of convergence exceeds sh.Taylor + exchanging sums:

ψ(t, y) =sX

m=0

am

∞Xk=0

y (k)(t)

k!mkhk − h

sXm=0

bm

∞Xk=0

y (k+1)(t)

k!mkhk

=

sX

m=0

am

!y(t) +

∞Xk=1

1

k!

sX

m=0

mkam − ksX

m=0

mk−1bm

!hky (k)(t).

Thus for having order p it is necessary and sufficient that

sXm=0

am = 0,sX

m=0

mkam = ksX

m=0

mk−1bm, k = 1, . . . , p,

sXm=0

mp+1am 6= (p + 1)sX

m=0

mpbm.

Let w = ez ; then w → 1 corresponds to z → 0. Taylor + exchanging sumsagain gives

ρ(z)− zσ(z) =∞X

k=0

1

k!

sX

m=0

mkam

!zk −

∞Xk=1

1

(k − 1)!

sX

m=0

mk−1bm

!zk .

Proof continues ..

Hence,ρ(z)− zσ(z) = czp+1 +O(zp+2)

for some c 6= 0 iff the conditions written above hold. The result follows by

restoring w = ez (remember that z = ln w = (w − 1) +O(|w − 1|2) for

w → 1).

An application to the 2-step Adams-Bashforth

Recall

yn+2 = yn+1 + h[3

2f (tn+1, yn+1)− 1

2f (tn, yn)].

An explicit computation shows that for ξ = w − 1

ρ(w)−σ(w) ln(w) = (ξ+ξ2)−(1+3

2ξ)(−1

2ξ2+

1

3ξ+. . . ) =

5

12ξ3+O(ξ4),

thus order 2 is validated. Exercise: try the same computation forthe 3-step Adams-Bashforth method.

Yeah! Happiness, joy!

We have a method for constructing consistent methods of arbitraryorder!

Bad news ... this sometimes does not help!

Consider the two-step implicit scheme

yn+2−3yn+1+2yn = h[13

12f (tn+2, yn+2)−5

3f (tn+1, yn+1)− 3

12f (tn, yn)].

Exercise: show that it has consistency order 2.Now consider the ODE y ′ = 0 with y(0) = 1 with unique solutiony ≡ 1. A single step reads

yn+2 − 3yn+1 + 2yn = 0,

whose general solution is given by yn = c1 + c22n, n = 0, 1, ....Suppose that c2 6= 0 and we need y0, y1 to initiate the iteration.The assumption c2 6= 0 is equivalent to y0 6= y1. Hence, in thiscase the method does NOT converge at all: in fact let t > 0 andh→ 0 and n→∞ so that hn→ t. Obviously n→∞ and|yn| → ∞, which is far from y(t) ≡ 1, ooops :-)

One smart student may say ...

... Ok but you have assumed a “stupid” choice of y0 6= y1. Wereyou choosing them equal, you would have ended up on the rightsolution, don’t you? Well, not really ... the motivation is thatalthough y0 = y1 in the first iteration, then after a while, due toroundoff errors yn 6= yn+1 and you are again dead in the water!

Touch to believe ...

Breakdown in the numerical solution of y ′ = −y , y(0) = 1, by anonconvergent numerical scheme, showing how the situationworsens with decreasing step size.

Dahlquist equivalence theoremWe say that a polynomial obeys the root condition if all its zerosreside in the closed complex unit disc and all its zeros of unitmodulus are simple.

Theorem (The Dahlquist equivalence theorem)

Suppose that the error in the starting values y1, y2, . . . ys1 tends tozero as h→ 0. The multistep method (16) is convergent if andonly if it is of order p ≥ 1 and the polynomial ρ obeys the rootcondition.

Touch to believe ... check that roots of

ρ(w) = w 2 − 3w + 2 = (w − 1)(w − 2)

are indeed w = 1, 2 and this explains the non-convergence of thecorresponding method. However, all Adam-Bashforth methods aresafe, because for them

ρ(w) = w s−1(w − 1).

Exercise!

Dahlquist first barrier

The multistep method (16) has 2s + 1 parameters and thanks tothe consistence order theorem we can tune them in order tooptimize the order (i.e., we can choose them to make it thelargest). The outcome, an (implicit) s-step method of order 2s, isunfortunately not convergent for s ≥ 3 (we have already seen thecase s = 3!). In general, it is possible to prove that the maximalorder of a convergent s-step method is at most 2b(s + 2)/2c forimplicit schemes and just s for explicit ones; this is known as theDahlquist first barrier.

Good construction

Choose an arbitrary s-degree polynomial ρ that obeys the rootcondition (how to do that?) and such that ρ(1) = 0 (how to dothat? Why do we need it?). Dividing the order condition by ln(w)

we obtain σ(w) = ρ(w)ln(w) +O(|w − 1|p). Suppose first p = s + 1

and expand the fraction in Taylor series around w = 1 and let σ bethe polynomial coinciding with the Taylor expansion up to orderO(|w − 1|s+1). The outcome is a convergent (!!!), s-step methodof order s + 1. Similarly , to obtain an explicit method of order s,we let σ be an (s − 1)th degree polynomial (to force bs+1 = 0)that matches the series to order O(|w − 1|s).

An example and Adams-Moulton methods

Try, for example, s = 2 and ρ(w) = w 2 − w . The solution isσ(w) = − 1

12 + 23 w + 5

12 w 2. for the implicit case, whereas for theexplicit case, where σ is linear we have p = 2, and so recover,unsurprisingly, the Adams-Bashforth scheme.The choice ρ(w) = w s−1(w − 1) is associated with Adamsmethods. We have already seen the explicit Adams-Bashforthschemes; their implicit counterparts are called Adams-Moultonmethods.

Explicit Runge-Kutta methods

Consider

y(tn+1) = y(tn) +

∫ tn+1

tn

f (s, y(s))ds

= y(tn) + h

∫ 1

0f (tn + hs, y(tn + hs))ds,

and replace the second integral by a (Gaussian) quadrature. Theoutcome might have been the method

yn+1 = yn + hν∑

j=1

bj f (tn + cjh, y(tn + cjh)), n = 0, 1, . . . ,

except that we do not know the value of y at the nodes tn + c1h,tn + c2h, . . . , tn + cνh. We must resort to an (additional)approximation!

Construction

We denote our approximation of y(tn + cjh) by ξj , j = 1, 2, . . . , ν.To start with, we let c1 = 0, since then the approximation isalready provided by the former step of the numerical method,ξ1 = yn. The idea is to construct the ξj ’s updating yn by a linearcombination of f (tn + c`h, ξ`) for ` < j . Specifically, we let

ξ1 = yn

ξ2 = yn + ha21f (tn, ξ1)

ξ3 = yn + ha31f (tn, ξ1) + ha32f (tn + c2h, ξ2)

. . . = . . .

ξν = yn + hν−1∑`=1

aν`f (tn + c`h, ξ`)

yn+1 = yn + hν∑`=1

bj f (tn, ξ`)

Butcher tableau

The matrix A = (aij)i ,j=1,...,ν , with zero entries where cofficientsare not used in the definition of the scheme, is called the RKmatrix, while the vectors b = (b1, . . . bν)T and c = (c1, . . . , cν)T

are the RK-weights and RK nodes respectively. We say that themethod has ν stages. A Runge-Kutta scheme is represented oftenby means of the so-called Butcher tableau

c | A−−− + −−−

| bT(17)

How to construct an explicit Runge-Kutta method?

Let us start assuming that

j−1∑i=1

aji = cj , j = 2, 3, , ν.

This is necessary and sufficient condition for a consistent methodto be invariant under autonomization (see [Lemma 3.19, F13]).The simplest device to derive RK methods (only valid for p ≥ 3!)consists of verifying the order for the scalar autonomous equation

y ′ = f (y), t ≥ t0, y(0) = y0.

How to construct an explicit Runge-Kutta method?

Let us start observing that Let show how to do it for ν = 3 only.We write below f ∼ f (tn) and y ∼ y(tn) unless differentlyspecified.

ξ1 = y

⇒ f (ξ1) = f

ξ2 = y + hc2f

⇒ f (ξ2) = f (y + hc2f ) = f + hc2fy f +1

2h2c2

2 fyy f 2 +O(h3)

ξ3 = y + h(c3 − a32)f (ξ1) + ha32f (ξ2)

= y + hc3f + h2a32c2fy f +O(h3)

⇒ f (ξ3) = f (y + hc3f + h2a32c2fy f ) +O(h3)

= f + hc3fy f + h2(1

2c2

3 fyy f 2 + a32c2f 2y f ) +O(h3).

How to construct an explicit Runge-Kutta method?

Therefore

yn+1 = y + hb1f + hb2(f + hc2fy f +1

2h2c2

2 fyy f 2)

+hb3[f + hc3fy f + h2(1

2c2

3 fyy f 2 + a32c2f 2y f )] +O(h4)

= yn + h(b1 + b2 + b3)f + h2(c2b2 + c3b3)fy f

+h3[1

2(b2c2

2 + b3c33 )fyy f 2 + b3a32c2f 2

y f ] +O(h4).

How to construct an explicit Runge-Kutta method?Since for a high-order consistent method we need

y ′(tn) ≈ f , y ′′(tn) ≈ ffy , y ′′′(tn) ≈ fyy f 2 + f 2y f ,

we have

y(tn+1) ≈ y + hf +1

2h2ffy +

1

6h3(fyy f 2 + f 2

y f ) +O(h4).

and comparing the terms of the exapansion up to the order threewith

yn+1 = yn + h(b1 + b2 + b3)f + h2(c2b2 + c3b3)fy f

+h3[1

2(b2c2

2 + b3c33 )fyy f 2 + b3a32c2f 2

y f ] +O(h4).

we obtain

b1+b2+b3 = 1, b2c2+b3c3 =1

2, b2c2

2 +b3c23 =

1

3, b3a32c2 =

1

6.

(18)

Classical RK methods

Third order three-stage explicit RK method:

0 |12 | 1

21 | −1 2

−−− + −−− −−− −−−| 1

623

16

(19)

and the so-called Nystrom scheme

0 |23 | 2

323 | 0 2

3−−− + −−− −−− −−−

| 14

38

38

(20)

Classical RK methods

The fourth order method is not beyond the capability of the Taylorexpansion technique presented, but it is significantly harder to besolved. The best-known fourth-order four-stage RK method is

0 |12 | 1

212 | 0 1

21 | 0 0 1

−−− + −−− −−− −−− −−−−| 1

613

13

16

(21)

To obtain order 5 we need six stages, and matters becomeconsiderably worse for higher orders.

Implicit RK methods

The idea behind implicit RungeKutta (IRK) methods is to allowthe vector functions ξ1, ξ2, . . . , ξν to depend upon each other in amore general manner. Thus, let us consider the scheme

ξj = yn + hν∑

i=1

aji f (tn + cih, ξi ), j = 1, 2, . . . , ν,

yn+1 = yn + hν∑

i=1

bj f (tn + cjh, ξj).

Here A = (aji )j ,i=1,...,ν is an arbitrary matrix (while before it wasstrictly lower triangular). We impose the convention

ν∑i=1

aji = cj , j = 1, 2, 3, , ν.

Disadvantage and advantages

The progressing of an implicit RK scheme one needs to solve ateach time a system of ν equations in the unknown ξj ’s to be ableto implement a step. This is clearly higher complexity, but IRKpossess important advantages; in particular they may exhibitsuperior stability properties (we shall see that later).

Unique solvability for IRK methods

Let us consider ki ’s so defined

ξj = y + hν∑

i=1

aji f (t + cih, ξi )︸ ︷︷ ︸:=ki

, j = 1, 2, . . . , ν.

Clearly we can define a set of equations for them

ki = f (t + cih, y + hν∑

j=1

aijkj), i = 1, 2, . . . , ν.

TheoremIf f is Lipschitz-continuous then for h > 0 small enoughk = (k1, . . . , kν), and hence ξ = (ξ1, . . . , ξν), are uniquely definedby the above system of nonlinear equations.

Unique solvability for IRK methodsSketch of a proof. Let us assume that f is globaly Lipschitzcontinuous with constant Λ. Set k = (k1, . . . , kν)T ∈ Rν×d and

Fi (k) = f (t + cih, y + hν∑

j=1

aijkj), i = 1, . . . , ν

F (k) = (F1(k), . . . ,Fν(k))T .

Let us consider the norm ‖k‖ = max1≤i≤ν ‖ki‖2. Thenk = (k1, . . . , kν)T of the Runge-Kutta step satisfies the fixed pointequation

k = F (k).

Now it holds

‖F (k)− F (k)‖ ≤ Λh max1≤i≤ν

ν∑j=1

|aij | ‖kj − k j‖2

≤ Λh‖A‖∞‖k − k‖.

For h < 1Λ‖A‖∞ the function F is a contraction with a unique fixed

point.

Larger time stepsOne of the advantages of IRK methods is their stability for largetime steps when dealing with stiff equations. The previous methodof computation is not so efficient, as it requires a barrier to h. Wecan therefore consider the following Newton iteration: Let usdefine zi = ξi − y , i = 1, . . . , ν

G (z) = z − h

∑ν

j=1 a1j f (t + cjh, y + zj)...∑ν

j=1 aνj f (t + cjh, y + zj)

and

z0 = 0

∇G (z`)∆z` = − G (z`), z`+1 = z` + ∆z`.

The drawback of the iteration is as usual that the Jacobi-Matrix∇G has to be evaluated and inverted at each step. A simplesolution is to substitute ∇G (z`) by ∇G (z0), giving the so-calledQuasi-Newton method.

Example of a two stages implicit Runge-Kutta method

Let us consider the method

ξ1 = yn +1

4h[f (tn, ξ1)f (tn +

2

3h, ξ2)]

ξ2 = yn +1

12h[3f (tn, ξ1) + 5f (tn +

2

3h, ξ2)]

yn+1 = yn +1

4h[f (tn, ξ1) + 3f (tn +

2

3h, ξ2)].

In short tableau form

0 | 14 −1

423 | 1

45

12−−− + −−− −−−

| 14

34

Try to prove that this method is of consistency order 3.

How can be construct Runge-Kutta methods? Usecollocation

Define the (unique!) polynomial u ∈ Pνu(tn) = yn

u(tn + cih) = f (tn + cih, u(tn + cih)), i = 1, . . . , ν,(22)

for 0 ≤ c1 < · · · < cν ≤ 1, the so-called relative nodes. Then onedefine the evolution

yn+1 = u(tn + h).

The evolution defined by (22) is called a collocation method

Indeed collocation methods are Runge-Kutta methods!

Let p1, . . . , pν be the Lagrangian polynomial associated to the(interpolation) nodes c1, . . . , cν (see [Section 1.1, ,F13]), so that

pj(ci ) = δij , i , j = 1, . . . , ν.

TheoremThe collocation method on the nodes c = (c1, . . . , cν) is animplicit Runge-Kutta method (b, c ,A) where

aij =

∫ ci

0pj(θ)dθ, i , j = 1, . . . , ν, (23)

and

bi =

∫ 1

0pi (θ)dθ, i = 1, . . . , ν. (24)

Recall how an implicit RK is formulated ...

Define

ki = f (tn + cih, yn + hν∑

j=1

aijkj), i = 1, 2, . . . , ν.

Then

yn+1 = yn + hν∑

i=1

biki .

ProofSet

kj = u(tn + cjh), j = 1, . . . , ν.

The Lagrange interpolation basis on c1, . . . , cν gives

u(tn + θh) =ν∑

j=1

kjpj(θ).

Integration over θ yields

u(tn + cih) = yn + h

∫ ci

0u(tn + θh)dθ

= yn + h

∫ ci

0

ν∑j=1

kjpj(θ)dθ

= yn + hν∑

j=1

[∫ ci

0pj(θ)dθ

]kj

= yn + hν∑

j=1

aijkj .

Proof continues ...

If we sustitute this expression in formula (22)

u(tn + cih)︸ ︷︷ ︸=ki

= f (tn + cih, u(tn + cih)︸ ︷︷ ︸=yn+h

Pνj=1 aijkj

),

we obtain

ki = f (tn + cih, yn + hν∑

j=1

aijkj) (25)

and analogously it follows

yn+1 = u(tn + h) = yn + h

∫ 1

0u(tn + θh)dθ

= yn + hν∑

i=1

biki .

(26)

We obtain so an implicit Runge-Kutta method.

Collocation as numerical quadrature ...

We define

y(tn + h) =

∫ tn+1

tn

f (t)dt, (27)

so thaty(t) = f (t), y(tn) = 0. (28)

The application of the collocation method will give us

y(tn+1) ≈ yn+1 = hν∑

i=1

biki

= hν∑

i=1

bi f (tn + cih),

(29)

over the nodes c1, . . . , cν , i.e., a Newton-Cotes integration formula!

Collocation as numerical quadrature ...

LemmaThe collocation method determined by the coefficients b, c, definesa quadrature

ν∑i=1

biϕ(ci ) ≈∫ 1=tn+h

0=tn

ϕ(x)dx , (30)

which is exact for polynomial of degree ν − 1.

Proof. It holds

θk−1 =νX

j=1

ck−1j pj(θ), k = 1, . . . , ν,

(since pj is a basis for Pν−1). Hence, it follows

νXj=1

bjck−1j =

νXj=1

Z 1

0

ck−1j pj(θ)dθ

=

Z 1

0

θk−1dθ =1

k, k = 1, . . . , ν,

and this means that polynomial of degree at most ν − 1 are exactly integrated.

Construction principles of collocation/IRK methods

Construction principle for ν-stages Runge-Kutta-methods:

I Choose a quadrature formula on nodes c1, . . . , cνI Construct the corresponding collocation methods by means of

bi =

∫ 1

0pi (θ)dθ, aij =

∫ ci

0pj(θ)dθ, i , j = 1, . . . , ν

Examples of collocation/IRK methods.Examples

I Gauss-methods:

I ci are the zeros of orthogonal polynomial w.r.t. the weightw ≡ 1 (Legendre polynomials, see [Sections 2.6.2 and 2.7.2,F13])

I ν-stages with consistency order p = 2νI they are A-stable (proof in the book of A. Iserles p. 63)I Example: the implicit middle point rule

yn+1 = yn + hf

(yn + yn+1

2

)I Radau-methods:

I cν = 1, c1, . . . , cν−1 zeros of the Jacobi-polynomial (see theRemark after [Section 2.6.2, F13])

I They are L-stableI Example: implicit Euler method

yn+1 = yn + hf (yn+1)

Not every implicit Runge Kutta method is a collocationmethod

Let ν = 2 and consider c1 = 0 and c2 = 23 . The corresponding two

stages collocation method build on these nodes has Butchertableau

0 | 0 023 | 1

313

−−− + −−− −−−| 1

434

But this is the unique collocation method associated to the nodesc1 = 0 and c2 = 2

3 . However, before we saw another two stagesimplicit Runge Kutta method with tableau (exercise!)

0 | 14 −1

423 | 1

45

12−−− + −−− −−−

| 14

34

We deduce that not all implicit Runge-Kutta methods can becollocation methods!

Remember the hard computation we did?

TheoremAn explicit RK method (c, b,A) has the consistency order p = 1 iff

1.P

i bi = 1 ,

the consistency order p = 2 iff additionally hold

2.P

i bi ci = 12,

the consistency order p = 3 iff additionally hold

3.P

i bi c2i = 1

3

4.P

i,j bi aijcj = 16,

and the consistency order p = 4, iff additionally hold

5.P

i bi c3i = 1

4

6.P

i,j bi ci aijcj = 18

7.P

i,j bi aijc2j = 1

12

8.P

i,j,k bi aijajkck = 124

Here the summations are between 1 and ν for ν ≥ p.

How can one solve this complicated system of nonlinearequations?!

We observe that the conditions 1,2,3, and 5 are easily recognizedas quadratue formulas

4∑i=1

biϕ(ci ) ≡∫ 1

0ϕ(t)dt,

which are exact for polynomials of degrees at most 3 on theinterval [0, 1]! We recall two important Newton-Cotes formulas,the classical Simpson formula and the so-called Newton- 3

8 -rule (see[Section 2.2.3 and Theorem 2.3, F13]).

Degree Name Rule Error

2 Simpson b−a6 (f0 + 4f1 + f2) − (b−a)5

2880 f (4)(ξ)

3 Newton- 38

b−a8 (f0 + 3f1 + 3f2 + f3) − (b−a)5

6480 f (4)(ξ)

Computing then ...

By the Newton - 38-rule we obtain the nodes c and weights b

c1 = 0, c2 =1

3, c3 =

2

3, c4 = 1

b1 =1

8, b2 =

3

8, b3 =

3

8, b4 =

1

8.

The rest of the conditions allows easily, together with the condition

j−1Xi=1

aji = cj , j = 2, 3, . . . , ν.

to fix all the values of the matrix A = (aij) (see pag. 74-75 in [F13]). Theresulting scheme

0 |13| 1

323| − 1

31

1 | 1 −1 1−−− + −−− −−− −−− −−−−

| 18

38

38

18

(31)

The classical 4 stages RK method revisited

The method (88)

0 |12 | 1

212 | 0 1

21 | 0 0 1

−−− + −−− −−− −−− −−−−| 1

613

13

16

can be computed analogously, but it is based on the Simpsonquadrature formula.

Stiff equations ... finally!

We consider the so-called Dahlquist test equation withcorresponding initial value

y = −λy , x(0) = 1, (32)

where λ > 0. The solution is explicitly given by y(t) = e−λt andmonotonically descresing to 0 for t →∞. When we apply theexplicit Euler method for this problem with time step h = 0.01, weobtain two different results, depending on λ.

What goes so wrong?!Let us consider the Euler method, defined by discrete evolution

yn+1 = yn + hf (yn),

applied to the Dahlquist test equation, to obtain

yn = (1− hλ)ny0.

In principle, we wish to expect that yn → 0 for n→∞ (since thisis the behavior of y(t) for t →∞). For this, it is necessary that

|1− hλ| < 1, and for h, λ > 0

− (1− hλ) < 1, (33)

and this holds iff

h <2

λ. (34)

But this is not satisfied for λ = 210 and h = 0.01! Hence, themethod is numerically unstable.

Implicit is betterLet us consider the implicit Euler method now:

yn+1 = yn + hf (yn+1), n = 0, 1, . . . , h > 0.

Again for the Dahlquist test equation it holds

yn+1 = xn − hλxn+1 or

yn+1 =1

1 + hλyn, n = 0, 1, . . . . (35)

We again require yn → 0 for n→∞ and we obtain the necessarycondition ∣∣∣∣ 1

1 + hλ

∣∣∣∣ < 1 (h, λ > 0)

⇐⇒ 1 < 1 + hλ (36)

⇐⇒ 0 < hλ

This is always satisfied for h, λ > 0! We conclude that the implicitmethod is always qualitatively correct, independently of the timestep!

Stiff problems ...

... are, roughly speaking, those for which explicit methods are“inefficient” for relatively large time steps.

Stability of solutions

Let us define Φt(y0) = y(t), the solution to the ODE

y ′ = f (y), y(0) = y0.

DefinitionA solution Φt(y0) is said

I stable, if for all ε > 0 there exists a δ > 0, such that

Φt(y) ∈ Bε(Φt(y0)) (37)

for all t ≥ 0 and for all y ∈ Bδ(y0).

I asymptotically stable, if additionally there exists a δ0 > 0,such that

limt→∞

∥∥Φt(y)− Φt(y0)∥∥ = 0 (38)

for all y ∈ Bδ0(y0).

I unstable, if it is not stable.

Stability of solutions

DefinitionA point y is a stationary point or a pixed point of the equationy ′ = f (y), if f (y) = 0. We call y asymptotically stable stationarypoint, if there exists δ0 > 0, such that

limt→∞

∥∥Φt(y)− y∥∥ = 0, (39)

for all y ∈ Bδ0(y).

Matrix exponential function

For A ∈ Cd×d , the function etA is defined by

exp(tA) := etA =∞∑

k=0

(tA)k

k!.

This series converges uniformly on intervals [−T ,T ]. The matrixexponential has the following properties

1. exp(tMAM−1) = M exp(tA)M−1, M ∈ GL(d);

2. exp(t(A + B)) = exp(tA) exp(tB), as soon as AB = BA;

3. For B = Block-diag(B1, . . . ,Bk)

exp(tB) = Block-diag(exp(tB1), . . . , exp(tBk)).

The proof is left to the brave students.

Linear equations

Consider the linear equation (here A is a matrix)

y = Ay . (40)

In this caseΦt(y0) = etAy0,

and this implies

Φt(y)− Φt(y0) = Φt(y − y0)

andsupt≥0

∥∥Φt(y)− Φt(y0)∥∥ ≤ ε

for ‖y − y0‖ sufficiently small, iff

supt≥0

∥∥Φt∥∥ = sup

t≥0

∥∥∥etA∥∥∥ <∞. (41)

Linear equations

We collect these observations in the following

LemmaA solution of the linear equation (40) is stable, iff (41) holds. It isasymptotically stable, iff

limt→∞

∥∥Φt∥∥ = 0.

Spectrum of matrices

Definition

I The spectrum σ(A) of a matrix A ∈ Cd×d is the set

σ(A) = λ ∈ C : det(λI − A) = 0

of all eigenvalues of A.

I The index ι(λ) of λ ∈ σ(A) is the dimension of theJordan-block of A associated to λ.

Main stability result

TheoremThe solution of y = Ay for A ∈ Cd×d is stable iff Re(λ) ≤ 0 for alleigenvalues λ of A and all eigenvalues for which Re(λ) = 0 havethe index ι(λ) = 1. The solution is aymptotically stable iffRe(λ) < 0 for all λ ∈ σ(A).

Sketch of the proofSince stability is an affine invariant we can use the 1. property of theexponential matrix, and assume A = J is already in Jordan form, i.e.,(J = MAM−1 and y = Ay implies (My) = J(My)). By property 3., we canadditionally reduce A to a single Jordan-Block J = λI + N, where

N =

0BBB@0 1 0

. . .

10 0

1CCCA| z

k

is a nilpotent matrix, i.e., Nk = 0. Then

‖exp(tA)‖ = |eλt | ‖exp(tN)‖

≤ eRe(λ)t

„1 + t ‖N‖+ · · ·+ tk−1

(k − 1)!‖N‖k−1

«| z

≤Mεeεt

,

for all ε > 0 and associated Mε > 0, from which

‖exp(tA)‖ ≤ Mεe(Re(λ)+ε)t

follows.

Sketch of the proof ... continues

Were Re(λ) < 0, then we could choose ε > 0 so small that Re(λ) + ε < 0 and‚‚‚etA‚‚‚ ≤ Mεe(Re(λ)+ε)t → 0, t →∞.

For Re(λ) = 0, k = 1, then ‖exp(tA)‖ = 1, and this means that the solution isstable. Were Re(λ) = 0 holding and k > 1, it would follow forek = (0, . . . , 0, 1)T ∈ Rk , that

exp(tJ)ek = eλt

„tk−1

(k − 1)!, . . . , t, 1

«T

,

and so ‖exp(tJ)‖ → ∞ for t →∞, and the condition ι(λ) = 1 is for all λ with

Re(λ) = 0 necessary. What happens when Re(λ) > 0 is left as an exercise.

Stablity of discrete evolutionFor one step methods the evolution is rules by a discrete fluxΨh(y) = y + hψ(y , h). For example for the explicit Euler-methodψ(y , h) = f (y) for y = f (y). Let us again consider

y = Ay ,

for which Ψh is also linear. In general:

yn+1 = Myn, n = 0, 1, 2, . . . (42)

DefinitionA linear iteration as (42) is called stable if

supn≥1‖Mn‖ <∞,

and asymptotically stable if

limn→∞

‖Mn‖ = 0. (43)

Stablity of discrete evolution

Assume we have y = Ay with continuous evolution

Φt(y) = exp(tA)y .

For one step methods, one wants Ψh(y) ' exp(hA)y . The issue isfor which h > 0 the discrete approximation is stable. For explicitmethods, in particular for ν-stages Runge-Kutta-methods, we have(exercise!)

Ψh(y) = P(hA)y ,

where P ∈ Pν . For implicit Runge-Kutta-methods (e.g., theimplicit Euler-method), it holds

Ψh(y) = R(hA)y ,

where R(z) = P(z)Q(z) is a rational function with polynomials P,Q

(where we assume that Q(λ) 6= 0 for all λ ∈ σ(A)).

Stablity of discrete evolution

TheoremThe linear iteration, given by M as in (42) is stable iff ρ(M) ≤ 1and all the eigenvalues λ for which |λ| = 1 have index ι(λ) = 1,where

ρ(M) := supλ∈σ(M)

|λ| .

Sketch of the proof. Similarly to the previous theorem: first,reduce the problem to a Jordan-block. Then it is

Jn = λnI + λn−1

(n

1

)N + · · ·+ λn−k+1

(n

k − 1

)Nk−1,

‖Jn‖ ≤ |λ|n(

1 + |λ|−1

(n

1

)‖N‖+ · · ·+ |λ|−(k−1)

(n

k − 1

)‖N‖k−1

)≤ Mθ(θ |λ|)n

for sufficiently large Mθ depending on θ > 1, so that θ |λ| < 1.The rest continues as in the proof of the previous theorem.

Summarizing

Continuous time Discrete timey = Ay yn+1 = exp(hA)yn

exp(hσ(A)) = σ(exp(hA))Reλ < 0 ∀λ ∈ σ(A) ⇐⇒ ρ (exp (hA)) < 1.

Rational functions ...

From the previous theorem, we saw that Ψh is stable iff ρ(Ψh) ≤ 1and ι(µ) = 1 for |µ| = 1, µ ∈ σ(R(hA)). Actually it holds

σ(R(A)) = R(σ(A)) (44)

for all rational functions R such that R(λ) 6=∞ for all λ ∈ σ(A).Hence, ρ(Ψh) = ρ(R(hA)) = maxλ∈σ(A) |R(hλ)| and we get

Ψhstable ⇐⇒

|R(hλ)| ≤ 1, for λ ∈ σ(A) and

ι(µ) = 1, for |µ| = 1.

Stability domains of rational functions

DefinitionThe set

S := SR := z ∈ C : |R(z)| ≤ 1 (45)

is called the stability domain of the function R.

Then we can say that Ψh is stable iff hλ ∈ SR for λ ∈ σ(A) andι(µ) = 1 for |µ| = 1 where µ = R(hλ), λ ∈ σ(A).

Examples of stability domains

For the explicit Euler method of discrete evolution

Ψh(y) = y + hf (y)

= y + hAy

= (I + hA)y = R(hA)y ,

e.g., R(z) = 1 + z . The associated stability domain is

SR = z ∈ C : |R(z)| ≤ 1= |1 + z | ≤ 1= B1(−1).

Examples of stability domains

For the implicit Euler method of discrete evolution

Ψhy = y + hA(Ψh(y))

= y + hAΨh(y),

implying Ψh(y) = (I − hA)−1y = R(hA)y , then R(z) = (1− z)−1.The associated stability domain is

SR = z ∈ C : |R(z)| ≤ 1

=

1

|1− z |≤ 1

= |1− z | ≥ 1

= (B1(1))c ,

the complement of B1(1).

Examples of explicit Runge-Kutta methodsFor consistent explicit Runge-Kutta methods of order s, we havethat

R(z) = 1 + z + · · ·+ 1

p!zs .

The corresponding stability domains (for s = 1, 2, 3, 4) are showedbelow.

Rational functions for Runge-Kutta methods

LemmaThe rational function R has for Runge-Kutta methods always theform

R(z) = 1 + zbT (I − zA)−1e, e = (1, . . . , 1)T . (46)

Moreover it is R(z) = P(z)Q(z) for unique polynomials P,Q ∈ Pν ,

P(0) = 1, Q(0) = 1.

ProofFor the Dahlquist equation y = λy , y(0) = 1, the application of RK gives

Ψh(1) = R(hλ) = 1 + hνX

j=1

bjλξj ,

ξi = 1 + hνX

j=1

aijλξj .

For z = hλ und ξ = (ξ1, . . . , ξν)T ∈ Rν we get

R(z) = 1 + zbT ξ (47)

ξ = e + zAξ, e.g. ξ = (I − zA)−1e. (48)

After using Cramer’s rule

Bx = c =⇒ xi =det(Bi )

det(B), Bi = (a1, . . . , c

i-th column, . . . , aν),

it holds

ξi =Pi (z)

det(I − zA)for Pi ∈ Pν−1, (49)

det(I − zA) ∈ Pν and R(z) = 1 +

Pνj=1 bjzPj

det(I − zA).

Example of an implicit Runge-Kutta methodThe trapezoidal rule

yn+1 = yn +h

2[f (tn, yn) + f (tn+1, yn+1)],

has Butcher scheme

0 | 0 01 | 1

212

−−− + −−− −−−| 1

212

The corresponding rational function is given by

R(z) = 1 + zbT (I − zA)−1e

= 1 + zˆ

12

12

˜„ 1 0− z

21− z

2

«−1 »11

–= 1 + z

ˆ12

12

˜ " 11+ z

21− z

2

#

= 1 +z

2

„1 +

1 + z2

1− z2

«=

1 + z2

1− z2

.

Stability concepts

We write C− = z ∈ C : Re(z) ≤ 0.

DefinitionA rational function R : C→ C such that C− ⊆ SR is calledA-stable.

TheoremA discrete evolution Ψh = R(hA) is (asymptotically) stable for allh > 0 and all (asymptotically) stable A if an only if R is A-stable.

Proof

Let Ψh be stable for all h > 0 and all stable A, this means that itholds |R(hλ)| ≤ 1 for all h > 0 and all λ ∈ C−, in particular|R(λ)| ≤ 1 for all λ ∈ C−. Hence, this imply the A-stability of R.Conversely let R be A-stable, h > 0 und A = λ ∈ C−. SinceC− ⊆ SR , it holds |R(hλ)| ≤ 1.Case 1: Reλ < 0. Then it holds hλ ∈ int S for all h > 0 andtherefore |R(hλ)| < 1.Case 2: Reλ = 0. Since A is stable, it follows ι(λ) = 1, henceι(R(hλ)) = 1. We conclude that Ψh is stable.

Stability concepts ... continues

I A rational function R such that C−α ⊆ SR is calledA(α)-stable, where

C−α = z ∈ C : |arg(−z)| ≤ α .

I An A-stable rational function R such that

limz→∞

R(z) = 0

is called L-stable. Were A asymptotically stable, thenΨh = R(hA)→ 0 for h→∞.

Examples of collocation/IRK methods.Examples

I Gauss-methods:

I ci are the zeros of orthogonal polynomial w.r.t. the weightw ≡ 1 (Legendre polynomials, see [Sections 2.6.2 and 2.7.2,F13])

I ν-stages with consistency order p = 2νI they are A-stableI Example: the implicit middle point rule

yn+1 = yn + hf

(yn + yn+1

2

)I Radau-methods:

I cν = 1, c1, . . . , cν−1 zeros of the Jacobi-polynomial (see theRemark after [Section 2.6.2, F13])

I They are L-stableI Example: implicit Euler method

yn+1 = yn + hf (yn+1)

Stability of multistep methods

Let us consider the application of a general multistep method as in(16), i.e.,

s∑m=0

amyn+m = hs∑

m=0

bmf (tn+m, yn+m), n = 0, 1, . . .

on the Dahlquist test equation, obtaining

s∑m=0

amyn+m = hλs∑

m=0

bmyn+m, n = 0, 1, . . .

or

s∑m=0

(am − hλbm)︸ ︷︷ ︸:=gm

yn+m =s∑

m=0

gmyn+m = 0, n = 0, 1, . . . . (50)

Stability of multistep methods

Defining the characteristic polynomial

η(w) =s∑

m=0

gmwm,

whose zeors are w1, . . . ,wq with multiplicities k1, k2, . . . , kq, where∑qi=1 ki = s, the general solution to (50) is given explicitly by

yn =

q∑i=1

ki−1∑j=0

cijnj

wni , n = 0, 1, . . . , (51)

the s constants cij are uniquely determined by the initial valuesy0, y1, . . . , ys−1.

Stability of multistep methods

LemmaLet us suppose that the zeros (as a function of w) of

η(z ,w) :=s∑

m=0

(am − zλbm)wm, z ∈ C,

are w1(z),w2(z), . . . ,wq(z)(z), while their multiplicities arek1(z), k2(z), . . . , kq(z)(z) respectively. The multistep method (16)is A-stable if and only if

|wi (z)| < 1, i = 1, 2, . . . , q(z)

for every z ∈ C−.

Proof

As for (51), z = λh,

yn =

q(z)∑i=1

ki (z)−1∑j=0

cijnj

wi (z)n, n = 0, 1, . . . ,

the behavior of yn depends on the magnitude of the wi (λh). If allreside inside the complex unit disc, then their power vanishes fasterthan any polynomial in n, hence yn → 0 for n→∞. Hence thecondition is sufficient for the A-stability to hold.However, if |w1(hλ)| ≥ 1, say, then there exist starting values suchthat c1,0 6= 0; therefore it is impossible for yn to tend to zero asn→∞. We deduce that the condition is necessary for A-stability.

Bad news! Stability domains of Adams methods

Backward differentiation formulae

An s-order s-step method is said to be a BDF if σ(w) = βw s forsome β ∈ R \ 0.

LemmaFor a BDF we have

β =

(s∑

m=1

1

m

)−1

and ρ(w) = β

s∑m=1

1

mw s−m(w − 1)m.

ProofBeing the order p = s one has

ρ(w)− βw s ln w = O(|w − 1|)s+1, w → 1.

We substitute v = w−1 and obtain

v sρ(v−1) = β ln v +O(|v − 1|)s+1, v → 1.

Now

ln v = ln(1 + [v − 1]) =sX

m=0

(−1)m−1

m(v − 1)m +O(|v − 1|)s+1

and we deduce that

v sρ(v−1) = βsX

m=0

(−1)m−1

m(v − 1)m,

hence

ρ(w) = βsX

m=1

1

mw s−m(w − 1)m.

To complete the proof, we need only to derive the explicit form of β. It follows

at once by imposing the normalization condition as = 1 on the polynomial ρ.

Examples and convergence

The simplest BDF has been already encountered: when s = 1 werecover the implicit Euler method! The next two BDFs are

s = 2 yn+2 − 43 yn+1 + 1

3 yn = 23 hf (tn+2, yn+2),

s = 3, yn+3 − 1811 yn+2 + 9

11 yn+1 − 211 yn = 6

11 hf (tn+3, yn+3),(52)

Since BDFs are derived by specifying σ, we cannot be sure thatthe polynomial ρ obeys the root condition. In fact, the rootcondition fails for all but a few such methods.

TheoremThe polynomial

ρ(w) = β

s∑m=1

1

mw s−m(w − 1)m

obeys the root condition and the underlying BDF method isconvergent if and only if 1 ≤ s ≤ 6.

Another derivation of BDF

We define the values yj+s byp(tj+k) = f (tj+s , yj+s)

p(tj+i ) = yj+i , i = 0, . . . , s.(53)

Via the Largange polynomials we can define

p(tj+s + θτ) =s∑

i=0

yi+jps−i (θ)

wherepi (−`) = δi`, i , ` = 0, . . . , s.

From (53) one obtains the explicit representation

αsyj+s + · · ·+ α0yj = hf (tj+s , yj+s) (54)

where αi = p′s−i (0), i = 0, . . . , s.

Connection to numerical differentiation

From (53) we have

p(ts) = f (ts , ys) and p(tj) = yj , j = 0, . . . , s,

and from (54)

p(ts) =αsp(ts) + · · ·+ α0p(t0)

h.

Hence, the formula

αsp(ts) + · · ·+ α0p(t0)

h' y(ts) (55)

can be used for the numerical differentiation of a function y ∈ C 1.This explains the name Backward Differentiation Formulae.

Stability domains of BDF methods

Stability domains of BDF methods of orders s = 2, 3, 4, 5, shown at the same

scale. Note that only s = 2 is A-stable.

Theorem (The Dahlquist second barrier)

The highest order of an A-stable a multistep method (16) is 2.

Additional complements: local error estimate and timestep refinement

As we know, for several methods we can hope for a global errorrate of the order of

err = O(hp).

But. how large is practically the constant hidden in the O? Isthere a way to ensure that the global eror keeps under a prescribedrelatively small bound?

Local error estimate and time step refinement

Let us consider now the solution of an ODE by means of aRunge-Kutta method of order p and simultaneously of aRunge-Kutta method of order p ≥ p + 1. We shall assume that yn

and xn are the corresponding numerical solutions and have

yn+1 = y(tn+1) + `hp+1 +O(hp+2),

xn+1 = y(tn+1) +O(hp+2)

The outcome of the subtraction of these equalities is

`hp+1 = yn+1 − xn+1,⇒ κ := ‖yn+1 − xn+1‖.

Once κ is available, we can check whether it reaches a wishedaccuracy threshold, for instance

κ ≤ δh.

If not, then it would better halve the time step h.

Local error estimate and time step refinement

Embedded Runge Kutta methodsA naive application of the above approach requires the doubling, atthe very least, of the expense of calculation, since we need tocompute both yn+1 and xn+1. This is unacceptable since, as a rule,the cost of error control should be marginal in comparison with thecost of the main scheme.However, when an explicit Runge-Kutta method is used for themain scheme it is possible to choose the two methods in such away that the extra expense is small. We shall choose the higherorder method

c | A−−− + −−−

| bT

such that ν ≥ ν,

c =

[cc

], A =

[A 0

A

]where c ∈ Rν−ν and A ∈ Rν−ν×ν , so that A is strictly lowertriangular.

Embedded Runge Kutta methods

In this case the vectors ξ1, . . . , ξν will be the same for both thelower and higher order method and only ξν+1, . . . , ξν−ν needs a bitof additional effort!We say then that the first RK method is embedded in the second.In tableau notation:

c | A−−− + −−−

| bT

−−− + −−−| bT

Embedded Runge Kutta methods

A well-known example of an embedded RK pair is the Fehlbergmethod, with p = 4, p = 5, ν = 5, ν = 6 and tableau

ode23. (Matlab). The Bogacki-Shampine pair embeds a 3rd order3-stage ERK into a 2nd order 4-stage ERK.ode45 (Matlab). The Dormand-Prince pair embeds a 5th order6-stage ERK into a 4th order 7-stage ERK.

Numerical methods for Partial Differential Equations: anintroduction

A first kind of partial differential equations (PDE) are the so-calledelliptic ones. We exemplify them by deriving the so-calledLaplace/Poisson equations from the elastic mechanics. Let usconsider an elestic membrane whose boundary has prescribe shape:

Which shape will the mambrane take? How does its shape dependson the one at the boundary?

Elastic energy

The membrane will certainly take the shape which minimizes itspotential energy. We define Ω ⊂ R2 the domain of the membraneand ∂Ω the boundary of the domain. Let u : Ω→ R the functiondescribing the displacement of the membrane with respect to itsideal position at rest and g : ∂Ω→ R the function describing thegiven shape of the membrane on the boundary. In particular weshall require that u ≡ g on ∂Ω and that the graph of u in factdescribes the surface of the membrane

We also define the potential elastic energy given by

J(u) =

∫Ω

√1 + |∂xu(x , y)|2 + |∂y u(x , y)|2dxdy

The variational problem and its simplificationWe search for a function u : Ω→ R which solves the followingvariational problem:

minu:Ω→R

J(u)

u|∂Ω = g .

To solve this problem, we first proceed with a semplification, underthe assumption that the minimal displacement function is “not toolarge”:

√1 + z = 1 +

1

2z +O(z2), z → 0,

leads us to the minimization of the approximate energy

J(u) =1

2

∫Ω

(|∂xu(x , y)|2 + |∂y u(x , y)|2

)dxdy

=1

2

∫Ω〈∇u,∇u〉dxdy → min!

Here the symbol 〈·, ·〉 represents the scala product in R2 and∇u = (∂xu(x , y), ∂y u(x , y))T is the gradient vector of u.

Calculus of variations

How can we minimize the energy J(u)? We compute the“derivative” of J and we set it to 0.Consider u + αv , α ∈ R, for v ∈ C 1(Ω), v |∂Ω = 0. The minimizer

u of J(u) should solve

0 =d

dαJ(u + αv)|α=0 = lim

α−>0

J(u + αv)− J(u)

α

= limα−>0

1

2α

ZΩ

h“|∂x(u(x , y) + αv(x , y))|2 + |∂y (u(x , y) + αv(x , y))|2

”−“|∂xu(x , y)|2 + |∂y u(x , y)|2

”idxdy

= limα−>0

1

2α

ZΩ

hα2“|∂xv(x , y)|2 + |∂y v(x , y)|2

”+ 2α〈∇u,∇v〉

idxdy

=

ZΩ

〈∇u,∇v〉dxdy .

Gauss divergence theorem

TheoremLet Ω ⊂ Rd be a compact set with a smooth boundary ∂Ω, andν : ∂Ω→ Rd is the external normal vector function, U ⊃ Ω is openon which we define a continuously differentiable vector fieldfunction F : U → Rd . Then it holds∫

Ωdiv Fdx =

∫Ω∇ · Fdx =

∫∂Ω〈F , ν〉ds.

Here we denoted div F (x) = ∇ · F (x) =∑d

j=1 ∂xj Fj(x) thedivergence of F = (F1, . . . ,Fd).

The Laplace equationLet’s go back now to our example and d = 2. We set F = v∇u,then div F = v∆u + 〈∇u,∇v〉, where

∆u = ∂2xxu + ∂2

yy y ,

is the Laplace operator. Hence, by the divergence theorem,∫Ω〈∇u,∇v〉dxdy = −

∫Ω

(v∆u)dxdy +

∫∂Ω

v〈∇u, ν〉s︸ ︷︷ ︸≡0

,

where the last term vanishes because v |∂Ω = 0. Hence∫Ω

(v∆u)dxdy = 0, ∀v ∈ C 1(Ω), v |∂Ω = 0,

or∆u = 0.

the Laplace or potential equation.

Solution on the disk

Let us remind that R2 ' C, z = x + iy = re iϕ. Assume that

w(z) = p(z) + iq(z),

is an holomorphic function(http://en.wikipedia.org/wiki/Holomorphic_function)i.e., it satisfies hte Cauchy-Riemann differential equations

∂xp = ∂y q, ∂y p = −∂xq.

This implies

∆p = ∂yxq − ∂xy q = 0,

∆q = −∂yxp + ∂xy p = 0,

Solution on the diskConsider Ω = B1(0) = z ∈ C : |z | < 1 and

w(x , y) = zk = r ke ikϕ = r k cos(kϕ)| z :=p(x,y)

+i r k sin(kϕ)| z :=q(x,y)

,

k = 0, 1, 2, . . . . This is an holomorphic function and hence

∆p = ∆q = 0.

If we consider the Fourier expansion of the boundary value function

g(x , y) = a0 +∞X

k=1

ak cos(kϕ) + bk sin(kϕ)

then

u(x , y) = a0 +∞X

k=1

r k (ak cos(kϕ) + bk sin(kϕ))

is a function solution of

∆u = 0

u|∂Ω = g .

Poisson equation

In Newtonian gravity and in Electrostatics, there is anotherimportant related equation

−∆u = f ,

called the Poisson equation, where f is a given datum and thesolution is usually additional constrained by suitable boundaryconditions, such as u|∂Ω = g .

Example from Newtonian gravity

In the case of a gravitational field F due to an attracting massiveobject, of density ρ, Gauss’ law for gravity in differential form canbe used to obtain the corresponding Poisson equation for gravity.Gauss’ law for gravity is:

∇ · F = −4πGρ.

Since gravity is a conservative field there has to exists Φ such that

F = ∇Φ.

By substitution we obtain

−∇ · ∇Φ = 4πGρ

or−∆Φ = 4πGρ.

The wave equationLet us start with a simplified model of longitudinal wave andconsider a longitudinal deformation as the one of a set of Nconnected springs subjected to a force:

The deviation of the right edge of the kth spring from its originalposition xk = x1 + kh, is denoted by uk(t). If we assume that allthe springs has the same rigidity/elasticity constantsK = k1 = k2 = k3 = . . . then the balance of forces gives

mu′′k = K (uk+1−uk)−K (uk−uk−1) = K (uk+1−2uk +uk−1). (56)

The wave equation...If we now assume that there is a continuous description limit forthe number N of springs to infinity, we have

uk(t) ≈ u(xk , t).

We re-write (56) as follows

m

h∂ttu(xk , t) = Kh

(u(xk+1, t)− 2u(xk , t) + u(xk−1, t)

h2

),

and for “h→ 0” (or for N →∞) – by rescaling aslo m(h)/h→ 1and K (h)h→ c2 – we obtain the wave equation:

∂ttu = c2∂xxu

which is usually complemented with the initial time conditions

u(x , 0) = f (x) ∂tu(x , 0) = g(x),

In general, the wave equation is given by

∂ttu = c2∆u

The heat equation

Let us consider a one dimensional body represented by an interval: theFourier’s law of heat diffusion says that the rate of flow of heat energy througha surface within the body is proportional to the negative temperature gradientacross the surface,

q = −k∇u (57)

where k is the thermal conductivity and u is the temperature. In onedimension, the gradient is an ordinary spatial derivative, and so Fourier’s law is

q = −k∂u

∂x

In the absence of work done, a change in internal energy per unit volume in thematerial, δQ, is proportional to the change in temperature, δu. That is,

δQ = cpρ δu

where cp is the specific heat capacity and ρ is the mass density of the material.By integration, choosing zero energy at absolute zero temperature, this can berewritten as

Q = cpρu.

The heat equationThe increase in internal energy in a small spatial region of the material[x − δx , x + δx ] over the time period [t − δt, t + δt] is given by

cpρ

Z x+δx

x−δx[u(ξ, t + δt)− u(ξ, t − δt)] dξ = cpρ

Z t+δt

t−δt

Z x+δx

x−δx

∂u

∂τdξ dτ

where the fundamental theorem of calculus was used. If no work is done andthere are neither heat sources nor sinks, the change in internal energy in theinterval [x − δx , x + δx ] is accounted for entirely by the flux of heat across theboundaries. By Fourier’s law (57), this is

k

Z t+δt

t−δt

»∂u

∂x(x + δx , τ)− ∂u

∂x(x − δx , τ)

–dτ = k

Z t+δt

t−δt

Z x+δx

x−δx

∂2u

∂ξ2dξ dτ.

By conservation of energy,Z t+δt

t−δt

Z x+δx

x−δx[cpρ∂τ − k∂ξξu] dξ dτ = 0.

This is true for any rectangle [t − δt, t + δt]× [x − δx , x + δx ], hence theintegrand must vanish identically:

cpρ∂τu − k∂ξξu = 0, or ∂τu =k

cpρpartialξξu,

which is the heat equation, where the coefficient α = kcpρ

is called the thermal

diffusivity.

The heat equation

In its full glory the heat equation reads

∂tu = α∆u,

and it is usually complemented by boundary and initial conditions

u(x , 0) = f (x),∀x ∈ Ω u(·, t)|∂Ω = g(·, t).

Solution of the heat equation in 1DThe following solution technique for the heat equation was proposed by JosephFourier in his treatise Theorie analytique de la chaleur, published in 1822. Letus consider the heat equation for one space variable. The equation isut = αuxx , where u = u(x , t) is a function of two variables x ∈ [0, L] and t ≥.Here we assume the initial condition

u(x , 0) = f (x) ∀x ∈ [0, L]

and the boundary conditions

u(0, t) = 0 = u(L, t) ∀t > 0.

Let us attempt to find a solution of the type

u(x , t) = X (x)T (t).

This solution technique is called separation of variables. Substituting u backinto the heat equation,

T (t)

αT (t)=

X ′′(x)

X (x).

Since the right hand side depends only on x and the left hand side only on t,both sides are equal to some constant value −λ. Thus:

T (t) = −λαT (t)

andX ′′(x) = −λX (x).

Solution of the heat equation in 1D ...Suppose that λ < 0. Then there exist real numbers B,C such that

X (x) = Be√−λ x + Ce−

√−λ x .

From the boundary conditions, we get X (0) = 0 = X (L) and thereforeB = 0 = C which implies u is identically 0. Suppose that λ = 0. Then thereexist real numbers B,C such that X (x) = Bx + C . From the boundaryconditions we conclude in the same manner as in 1 that u is identically 0.Therefore, in order to have nontrivial solutions, it must be the case that λ > 0.Then there exist real numbers A,B,C such that

T (t) = Ae−λαt

andX (x) = B sin(

√λ x) + C cos(

√λ x).

From the boundary condition we get C = 0 and that for some positive integern, √

λ = nπ

L.

This solves the heat equation in the special case that the dependence of u hasthe given special form. In general, the solution is given by

u(x , t) =∞Xn=1

Dn sin“nπx

L

”e− n2π2αt

L2

where Dn = 2L

R L

0f (x) sin

`nπxL

´dx .

Finite Difference Method for Elliptic EquationsThe main idea is to approximate derivatives by means of finitedifferences.Let us start with a simple 1D boundary value problem. LetΩ = (0, 1) and we consider u : Ω→ R solution of the Poissonequation

u′′ = f , on Ω,

u(0) = g0

u(1) = g1.

We construct the numerical approximation to the solution on agrid Ωh := h, 2h, . . . , 1− h and we define Ωh = Ωh ∪ 0, 1 ofgrid size h = 1

n , for some n ∈ N.Exercise: show that

u′′(x) =u(x + h)− 2u(x) + u(x − h)

h2+ h2R, (58)

where |R| = |R(x)| ≤ 112‖u

(4)‖∞, as soon as u ∈ C 4(Ω).

Discrete boundary value problemAccording to the approximation derived from (58), we define thefollowing discrete boundary value problem:

uh(x + h)− 2uh(x) + uh(x − h)

h2= f (x), for all x ∈ Ωh,

uh(0) = g0

uh(1) = g1.

Our hope is that uh ≈ u on Ωh. The discrete linear equations canbe written in matrix form as follows

1

h2

0BBBB@−2 1 0 . . . . . . 01 −2 1 0 . . . 0. . . . . . . . . . . . . . . . . .0 . . . 0 1 −2 10 . . . . . . 0 1 −2

1CCCCA| z

:=Lh

0BBBB@uh(h)

uh(2h). . .

uh(1− 2h)uh(1− h)

1CCCCA| z

:=uh

=

0BBBB@f (h)− 1

h2 g0

f (2h). . .

f (1− 2h)f (1− h)− 1

h2 g1

1CCCCA| z

:=fh

Hence uh can be computed by solving the above defined linearsystem by using methods from numerical linear algebra (Gausselimination, LU/QR decompositions, Gauss/Jacobi methods,gradient and conjugate gradient methods, multigrid methods).

Abstract finite differencesLet us consider a general elliptic problem defined on a openbounded domain Ω ⊂ R2

Lu = f on Ω, u|∂Ω = g , (59)

where L is an elliptic differential operator, for instance L = −∆.We shall consider a discretization of the problem on a grid Ωh ⊂ Ωof grid size h, for instance Ωh = Ω∩ hZ2. We define ∂Ωh ⊂ Ω as afinte set of boundary grid points approximating ∂Ω and being closeto it. We finally write Ωh = Ωh ∪ ∂Ωh.An abstract finite difference method for the solution of the abovewritten equation is given by means of the representation

Lu = Lhu + error ,

whereLhu(x) =

∑x ′∈N(x)

w(x , x ′)u(x ′), x ∈ Ωh,

and N(x) ⊂ Ωh is the set of neigborhood points of x ∈ Ωh,w(x , x ′) = O(h−2).

Discrete boundary value problem

Accordingly we define the discrete boundary value problem

Lhu = fh on Ωh, uh = gh on ∂Ωh, (60)

where fh, gh are approximations to f and g on Ωh and ∂Ωh

respectively, for instance f ≡ fh and g ≡ gh.

Consistency error

Denoted by δh(x) = Lhu(x)− fh(x) the consistency deviation,provided u being the exact solution of the boundary value problem(59), we say that the finite difference method is consistent if

limh→0‖δh‖∞ = 0.

It has the consistency order p if

‖δh‖∞ = O(hp), h→ 0.

Notice that this is different definition with respect to theconcistency error for methods for ODE, since there we requiredthat the consistency error was of order O(hp+1)!