2. Overview of numerical analysis

numerical calculus ● linear algebraic systems ● interpolation ● ordinary differential equations ●

relaxation

1. System of linear algebraic equations

nnnnnn

n

n

b

bb

x

xx

aaa

aaaaaa

2

1

2

1

21

22221

11211

bAxOrder n

• The execution time of Gaussian elimination depends on the amount of floating point operations (FLOPS).

• Cramer’s rule ~ (n+1)!• Say for a 99 X 99 matrix in a 2 GHz computer,

– Gaussian elimination: FLOPS ~ 3.5 x105, t = 1.8 x 10-4 s

– Cramer’s rule: FLOPS ~ 100! = 9.3 x 10157, t = 1.3 x 10142 years!

][3

2222 231

1

221

1

nOnkknnnknknGn

k

n

k

1.1 Band systems (Thomas method)• In typical finite difference or element schemes, band

systems occur often.

n

n

nnnn

nnnnnn

bb

bb

xdxaxcxdxd

xcxdxaxcxdxa

xcxd

1

2

1

11

111

433322

322211

2111

Instead of storing n × n matrix,we need only to store a, d and c

nnnn

n

b

bb

x

xx

dac

dacda

cd

2

1

2

1

1

1

32

221

11

0000

000

n

n

nnnn

nnnnnn

bb

bb

xdxaxcxdxd

xcxdxaxcxd

xcxd

1

)1(2

1

11

111

433322

322)1(

2

2111

pivoting

11

12

)1(21

1

12

)1(2 , b

da

bbcda

dd

n

n

nnnn

nnnnnn

bb

bbb

xdxaxcxdxd

xcxdxcxd

xcxd

1

)2(3

)1(2

1

11

111

433)2(

3

322)1(

2

2111

)1()1(1

)(1)1(1

)(1 ,

kkk

k

kk

kkkk

k

kk

kk b

da

bbcd

add

Similarly

pivoting

k = 1,…,n - 1

)1(

)2(1

)2(3

)1(2

1

)1(11

)2(1

433)2(

3

322)1(

2

2111

nn

nn

nn

n

nnnn

n

bb

bbb

xdxcxd

xcxdxcxd

xcxd

Until ...

Then use back substitution

1,,1)1(1

)1(

nkd

xcbx k

k

kkk

kk

1.1.1 Thomas method example

114152472322

43

432

321

21

xxxxx

xxxxx

111572

1400241002310021

111552

1400241002100021

111052

1400220002100021

91052

3000220002100021

pivot

01

)1)(2(211

)2)(2(522

)3)(2(10339

1234

xxxx

1.2 Gauss-Seidel method

nnnnnn

n

n

b

bb

x

xx

aaa

aaaaaa

2

1

2

1

21

22221

11211

)(1

)(1

)(1

11,2211

2232312122

2

1131321211

1

nnnnnnnn

n

nn

nn

bxaxaxaa

x

bxaxaxaa

x

bxaxaxaa

x

)(1

)(1

)(1

)1(11,

)1(22

)1(11

)1(

2)(

2)(

323)1(

12122

)1(2

1)(

1)(

313)(

21211

)1(1

nk

nnnk

nk

nnn

kn

knn

kkk

knn

kkk

bxaxaxaa

x

bxaxaxaa

x

bxaxaxaa

x

1.2.1 Gauss-Seidel method example

52014

19024011

1364

3

2

1

xxx

5000.002500.002500.0

05556.0011111.002222.0

21875.001563.004688.0

)1(2

)1(1

)1(3

)1(3

)1(1

)1(2

)(3

)(2

)1(1

kkk

kkk

kkk

xxx

xxx

xxx

k = 0 k = 1 k = 2 k =3x1 0.21875 0.22916 0.22955 0.22955x2 0.05556 0.06621 0.06613 0.06613x3 0.50000 0.49262 0.49261 0.49261

1.2.2 Convergence criterion

n

ijijii aa

1

• FLOPS for Gauss-Seidel ~ 4n2 (k+1) , k depends on accuracy required obviously

• For a 999 X 999 matrix in a 2 GHz single-processor computer, – Gaussian elimination: FLOPS ~ 3.3 x 108, t = 0.167 s– Gauss-Seidel method: FLOPS (say 5 iterations) ~ 2.4

x 107, t = 0.0121 s

1.3 Relaxation

To accelerate the Gauss-Seidel iteration in Ax = B, we can consider

)()1()1(

1

)(1

1

)1()1(

)1(

1

mi

mi

mi

n

ij

mjij

i

j

mjiji

ii

mi

xzx

xaxaba

z

Relaxation parameter

2. Interpolation

nixfxP ii ,...,1][][

• Interpolation is NOT curve-fitting!

2.1 Lagrange polynomials

1

0

1

01

][

1,...,1,0][][][

n

ij ji

ji

n

iiin

xxxx

xl

nixlxfxPLagrange polynomial

!][)(][][

)(1

01 n

fxxxPxfnn

i

nin

Lagrange error

• Suppose j (0 j n-1), lj:

• Then

• To satisfy this, xi (i j) must be root of lj

• Since lj[xj] = 1, thus

kjkj

xl kj

10

][

1

01 ][][][

n

iiin xlxfxP

1

0

1110

)(

)())(()(][n

jiii

njjii

xxC

xxxxxxxxCxl

1

0

1n

ji ii xx

C

2.2 Example

2sin1sin0sin][210210

i

i

xfxi

2)1()2(sin)2()1(sin

2)2)(1()0(sin][2

xxxxxxxP

2)1(

)12)(02()1)(0(][

)2()21)(01()2)(0(][

2)2)(1(

)20)(10()2)(1(][

2

1

0

xxxxxl

xxxxxl

xxxxxl

2.3 Pitfalls of interpolating polynomials

222

1][xba

xf

3. Numerical differentiation: 2-point formula

hxfhxfxf

hxfhxfxf

h

][][]['

][][lim]['

000

0

Two-point formula

][][''2

][][]['

][''2

]['][][

000

2

000

hOfhh

xfhxfxf

fhxhfxfhxf

3.1 Error of 2-point formula

nx hxexf 101][ 0

n f[1+h_k] f[1+h_k]-e f'[x]1 3.004166024 0.2858841600 2.85884162 2.745601015 0.0273191870 2.73191873 2.721001470 0.0027196410 2.71964104 2.718553670 0.0002718420 2.71842005 2.718309012 0.0000271830 2.71830006 2.718284547 0.0000027180 2.71800007 2.718282100 0.0000002720 2.72000008 2.718281856 0.0000000270 2.70000009 2.718281831 0.0000000003 3.0000000

10 2.718281829 0.0000000000 0.0000000

3.2 Error analysis of 2-point formula

hhM

h

xfxfxfxfhM

hxfxf

hxfxf

hxfxfxf

hxfxfxf

22

][][][][

2

][][][][][][]['

][][]['

2

00112

0101010

010

3.3 Step-size optimisation

n

n

n

nMnh

hnnMhhQ

21

optimal ][)!1(

)!1(][][

02587.01

)105.0(6

1105.02cos][

61

10

optimal

10

h

Mnxxf

3.4 Interpolatory differentiation

201

1

211

0210

12

0111

011

1010

110

1101

1012

01010

2)()(][

)()(][2

)()(][]['

))(())((][

))(())((][

))(())((][][

hxxxxxf

hxxxxxf

hxxxxxfxP

xxxxxxxxxf

xxxxxxxxxf

xxxxxxxxxfxP

hxxhxxx

][62

][][]['

))()((6

][][][

][][2][][''

2][][]['

)3(2

000

101

)3(

2

2000

0

000

fhh

hxfhxfxf

xxxxxxfxPxf

hhxfxhxfxf

hhxfhxfxf

3.5 Method of undetermined coefficients

][3

2]['''3

4][''2]['2][]2[

][24

]['''6

][''2

]['][][

2)4(

4

0

3

02

000

1)4(

4

0

3

0

2

000

fhxfhxfhxhfxfhxf

fhxfhxfhxhfxfhxf

]2[][][]['' 0000 hxcfhxbfxafxf

32

)3(1

)3(

02

00

000

][34][

6

][''22

][')2(][)(

]2[][][

hfcfb

xfhcbxhfcbxfcba

hxcfhxbfxaf

• Determine derivatives by comparing coefficients to the desired degree of accuracy

• For f’

• Similarly for f’’h

hxfhxfxfxf

hc

hb

ha

cbh

cbcba

6]2[][4][3]['

61,

32,

21

022

,12,0

0000

2000

0 3]2[][2][3][''

hhxfhxfxfxf

3.6 Finite difference formulae

• Central difference

• Forward / Backward difference

][][][2][][''2

][][][' 22

1111 hOh

xfxfxfxfh

xfxfxf iiii

iii

][][2][5][4][][''2

][3][4][]['2

2123

12

hO

hxfxfxfxfxf

hxfxfxfxf

iiiii

iiii

4. Newton-Cotes integration formulae

b

a

n

iii xfwdxxf

0

][][

b

a

n

i

b

a ii

n

ii

nn

iii

dxxlxfdxxf

xxn

fxlxfxf

0

0

)1(

0

][][][

)()!1(

][][][][

4.1 Trapezoidal rule

1

01

1

01

111

1

11

1

11

][][2

][][

][][2

][][2

][

][][][

1

1

n

iii

n

i

x

x

b

a

iiiiii

x

x

ii

ii

ii

ii

xfxfhdxxPdxxf

xfxfhxfxfxxdxxP

xxxxxf

xxxxxfxP

i

i

i

i

Elemental trapezoidal rule

Composite trapezoidal rule

12][)(

12][error total

12][

!2][))((errorn integratio

!2][))((1 toerror ion interpolat

)2(21

0

)2(3

)2(3)2(

1

)2(

1

1

fhabfh

fhdxfxxxx

fxxxxii

n

i

x

x ii

ii

i

i

• FUNCTION Trap (h,n,f)– Sum = f_0– DO i = 1, n-1

• Sum = sum + 2*f_i– END DO– sum = sum + f_n– Trap = h * sum /2

• END Trap

4.2 Simpson’s 1/3 rule

])[][4][2

][4][2][4][(3

][

])[][4][(3

][

][

][][][

12

3210

212

12

1

22

21

2

11

2

2

1

12

2

nnn

b

a

iii

x

x

ii

i

ii

ii

ii

i

ii

ii

ii

i

ii

ii

xfxfxf

xfxfxfxfhdxxf

xfxfxfhdxxP

xxxx

xxxxxf

xxxx

xxxxxf

xxxx

xxxxxfxP

i

i

Composite Simpson’s rule

Elemental Simpson’s rule

][180

)(error total

][90

2 to fromerror

)4(4

)4(5

fhab

fhii

• FUNCTION Simpint(a,b,n,f)– h = (b-a)/n– sum = sum + Simp13(h,m,f)– Simpint = sum

• END Simpint

• FUNCTION Simp13(h,n,f)– Sum=f(0)– DO i = 1,n-2,2

• Sum = sum + 4*f_i + 2*f_(i+1)– END DO– sum = sum + 4*f_(n-1) +f_n– Simp13 = h * sum /3

• END Simp13

• Not much improvement gained

][80

)(error total

][6480

3 toerror

])[][3

][3][2][3][3][(83][

])[][3][3][(83][

3

)4(4

)4(5

1

43210

32133

fhab

fhii

xfxf

xfxfxfxfxfhdxxf

xfxfxfxfhdxxP

n

nn

b

a

iiii

x

x

i

i

4.3 Simpson’s 3/8 rule

4.4 Gauss-Legendre quadrature

1

0

][][][n

kkk

b

a

b

axPwdxxPdxxf

b

a

n

iii xfwdxxf

0

][][

:objective

Problem: Find xk and corresponding wk for various n so that error is minimum

even1

2odd0

1)1(1

0

2

1,1][let

11

10

1

1

1

0

1

1

1

0

0

0

kk

k

kdxxxw

dxxxw

dxxxw

baxxP

kk

n

i

kii

n

iii

n

iii

jj

]0[2][

0,20,21

1

1

00

101

000

fdxxf

xwxwxwn

343.2

]3

1[]3

1[][

31,

31,1

0,32

,0,22

1

1

1

1

1010

311

300

211

200

111

100

011

000

dxe

ffdxxf

xxww

xwxwxwxw

xwxwxwxwn

x

4.4.1 Gauss-Legendre nodes

Points Weight Arguments1 1 02 1 ±0.5773502693 0.5555556 ±0.774596669

0.8888889 04 0.3478548 ±0.861136312

0.6521452 ±0.3399810445 0.2369269 ±0.906179846

0.4786287 ±0.538469310

0.5688889 06 0.1713245 ±0.932469514

0.3607616 ±0.661209386

0.4679139 ±0.238619186

4.4.2 Gaussian limit-transform

n

iii

b

a

b

a

abxabfwabI

dtabtabfabI

baba

baba

dttfIxt

dxxfI

0

1

1

222

222

,21,1

][1

][

4.4.3 Example

793.4exact 793.4

53,0,

53

95,

98,

95

25.22

25.02

5.2,0.2

210210

5.2

0.2

II

xxxwww

abab

badxe x

4.4.4 Characteristics of Gauss-Legendre quadrature

• Advantages:– Can handle singular integrals– Very small number of points to give very accurate

results– Simple calculations

• Disadvantages:– Complicated theories– Use of table for weights and nodes– Difficult to estimate errors– Unable to use preceding formulae: inefficiency

5. Ordinary differential equations

• Initial-value problems (IVP): ODE in which information about the solution is given at a single time point, e.g. first-order systems

• Boundary-value problems (BVP): ODE in which information about the solution is given by two or more points, e.g. multi-order systems

00 ][]],[,[][' yxyxyxfxy

],[':]['][

ionconsideratunder domain ])[,(

00

yxfyxyyxy

xyx

];,[1 hyxhyy iiii

Increment function

5.1 Euler’s method

• No differentiation needed

• Very simple algorithm'

1

1 jjj hyyym

5.1.1 Example

001.0],[

0],[0]0[,'

1112

0001

22

yxhfyyyxhfyy

yyxy

0

0.1

0.2

0.3

0.4

0 0.2 0.4 0.6 0.8 1

x

y

exact

euler

x_j y_j0.0 0.0000000.1 0.0000000.2 0.0010000.3 0.0050000.4 0.0140030.5 0.0300220.6 0.0551120.7 0.0914160.8 0.1412520.9 0.2072471.0 0.292542

5.2 Foundations of Runge-Kutta’s method

• Consider the Taylor’s series for m = 2

• Now consider a small step h* (smaller than h)],[]][,[][''

!3''

!2' 2

1

jjjjjj

jjjj

yxfxyxfxyy

hy

hy

yy

*

**

*

**

*

*

]],[]],[][,[

]][,[]][,[

][']['][''''

hyxfyxfhxyhxf

hxyxfhxyhxf

hxyhxy

xyy

jjjjjj

jjjj

jjjj

• Let h* = h

• Comparing

]],[,[

21],[

2111 jjjjjjjj yxhfyhxfyxfyy

22111

21

121

21

211

],[],[

kkhyy

hkyhxfkyxfk

jj

jjjj

5.2.1 General Runge-Kutta’s formula

sikhyhxfk

yxfk

khyy

i

mmimjiji

jj

s

iiijj

,,2,

],[

generalin

1

1

1

11

stage

5.2.2 Euler’s method and its modification

1 when minimizederror

],[1

1

11

jjjj yxfhyys

12

1

21

21221

2,

2

],[

21

2110

2

khyhxfk

yxfk

hkyy

s

jj

jj

jj

Euler’s method

Modified Euler’s methodHeun’s method

5.2.3 Heun’s and Ralston’s method

12

1

211

32,

32

],[4

34

2

hkyhxfk

yxfk

kkhyy

s

jj

jj

jj

12

1

211

43,

43

],[3

23

2

hkyhxfk

yxfk

kkhyy

s

jj

jj

jj

Heun’s 2nd method

Ralston’s method

5.2.4 Classical 4th-order Runge-Kutta’s method

32

23

12

1

43211

,21,

21

21,

21

],[

226

hkyhxfk

hkyhxfk

hkyhxfk

yxfk

kkkkhyy

jj

jj

jj

jj

jj

• SUB Rk4(x,y,h,ynew)– CALL Derive(x,y,k1)– ym = y+k1 * h/2– CALL Derive(x+h/2,ym,k3)– ym = y + k2 * h/2– CALL Derive(x + h/2, ym, k3)– ye = y + k3 *h– CALL Derive (x+h,ye,k4)– slope = (k1+2(k2+k3)+k4)/6– ynew = y + slope *h– x = x+h

• END SUB

5.2.5 Errors of various RK’s method

5.2.6 Runge-Kutta’s method example

i xi yi k1 k2 k3 k4 yi+10 0 0 0 0.0025 0.002500016 0.010000063 0.0003333351 0.1 0.000333335 0.010000111 0.022500694 0.022502127 0.040006675 0.0026668752 0.2 0.002666875 0.040007112 0.062521783 0.062533558 0.090079571 0.0090034983 0.3 0.009003498 0.090081063 0.122682454 0.122729148 0.160452686 0.0213594474 0.4 0.021359447 0.160456226 0.203363317 0.20349399 0.251739628 0.0417912885 0.5 0.041791288 0.251746512 0.305457034 0.305756316 0.365236971 0.0724481256 0.6 0.072448125 0.365248731 0.430728406 0.431333095 0.503359068 0.1156603057 0.7 0.115660305 0.503377306 0.582332855 0.583460365 0.670278207 0.1740810048 0.8 0.174081004 0.670304196 0.765596188 0.767597115 0.872921065 0.2509078699 0.9 0.250907869 0.872954758 0.989263005 0.992722749 1.122626133 0.350233742

10 1 0.350233742 1.122663674 1.267634078 1.273577737 1.438093656 0.477620091

0]0[,' 22 yyxy

5.2.7 Runge-Kutta’s parameter

• The central principle of Runge-Kutta approach is to choose the parameters i, i and im for fixed s so that a power series expansion in h agrees with that of Taylor’s expansion for as high a power of h as possible.

• Almost impossible to solve, trial and error

s

iiijj khyxg

1

],,[Define

1

00

)(

!],,[

!][ p

q

q

qjj

q

j

p

q

qjq

qh

dhhyxgd

hyhq

xy

5.3 Predictor-corrector algorithms

• Use more previous or intermediate data to predict values

5.3.1 Adams-Bashforth-Moulton formulae

Adams-Bashforth predictor Adams-Moulton corrector• not self-starting, need Runge-Kutta to obtain

first few yj

p

mmjmjm

p

mmjmj yxfhyy

011

111 ,

• Obtain yj,…,yj+1-i by Runge-Kutta formula• Use Adams-Bashforth predictor to obtain yj+1 (=

yj+1I)

• Define f[xj+1,yj+1I]

• Use Adams-Moulton to construct yj+1II with

f[xj+1,yj+1I]

• Compute f[xj+1,yj+1II]

• Repeat until converge

5.3.2 ABM coefficients

p

mmjmjmjj yxfhyy

0111 ,

720251

7201276

7202616

7202774

7201901

249

2437

2459

2455

125

1216

1223

21

23

54321

5432

11

p

72019

720106

720264

720646

720251

241

245

2419

249

121

128

125

21

21

43210

5432

11

p

Adams-Bashforth predictor Adams-Moulton corrector

5.3.3 Example

],[21],[

21

],[21],[

23

2],['

1

1

11

11

jjC

jjj

jjjjjC

yxfyxfhyy

yxfyxfhyy

pyxfy

j

jAdams-Bashforth

Adams-Moulton

5.4 System of ODE

• All methods discussed can be extended to the system shown above with no apparent difficulty.

n

n

n

yyyxfdxdy

yyyxfdxdy

yyyxfdxdy

,...,,,

,...,,,

,...,,,

2114

2112

2111

5.5 Higher-order BVP

• Decomposing the ODE to a system of 1st-order ODE• Introduce the new variables yi

10][

]][,],[,[][)(

00)(

)1()(

kjyxy

xyxyxfxyjj

kk

][][

],['][],[][

)1(

2

1

xyxy

xyxyxyxy

kk

)1(001

)2(0011

)1(00232

)0(00121

][]],[,],[,[]['

][],[]['

][],[]['

][],[]['

kkkk

kkkk

yxyxyxyxfxy

yxyxyxy

yxyxyxy

yxyxyxy

1(0

)0(0

0

21

]][,[

][

]['

]['

kk y

y

xyxf

xy

xy

xy y

• Solve as a system of 1st-order ODE

5.6 Shooting method

• Based on converting the BVP into an IVP• Solve by trial and error

• Define200]10[,40]0[:0)20(01.0 TTT

dxdT

)20(01.0

Tdxdz

zdxdT

• Make guess

• Therefore, looking at the boundary conditions, a solution must exist between 0 and 10

• Solve it using any rootfinding techniques, eg secant method

• Continue until converge

[0] 20 [10] 285.8980[0] 10 [10] 168.3797

z Tz T

6907.12)3797.168200(3797.1688980.285

102010]0[

z