Solving Linear Systems:Iterative Methods · (PDE systems solutions) – convergence guaranteed if...

Solving Linear Systems:Iterative Methods

� Motivation

� Jacobi Iteration

� Gauss Seidel Iteration

� Successive Over Relaxation

� Determinants

� Matrix Inversion

� Analysis

ITCS 4133/5133: Intro. to Numerical Methods 1 Iterative Methods

Motivation

◦ Consider the 2D potential field equation defined over a mesh ofpoints by a partial differential equation(PDE):

uxx + uyy = 0

given the boundary values and a mesh with ∆x = ∆y = 0.25.

◦ These systems are large and very sparse, and thus more appropri-ate for iterative solutions.


Iterative Methods

� Solution of linear systems using a trial and error approach.

� Assume a set of unknowns, and successively refine the estimates.

� Can produce exact solution; precision dependent on number of iter-ations.

� Major Advantage: Can be used to solve non-linear equations, notpossible using direct (elimination) approaches.


Iterative Methods: Overview

� To solve Ax = B, convert to x = Cx + d and generate a set ofapproximations, x(1),x(2), . . . ,x(n), where

x(k) = Cx(k−1), +d

� Three common techniques, Jacobi, Gauss-Seidel, SOR.


Jacobi Iteration

� Must start iteration with initial estimates for the Xis

� After computation of Xis, i = 1, 2, · · · , n, use the new estimates forthe next iteration.

� Convergence: Successive estimates are acceptably small.


Jacobi IterationConsider a linear system:

a11X1 + a12X2 + · · · + a1nXn = C1

a21X1 + a22X2 + · · · + a2nXn = C2... ...... ...

an1X1 + an2X2 + · · · + annXn = Cn

Solve for each unknown, Xi

X1 =C1 − a12X2 − a13X3 − · · · − a1nXn

a11

X2 =C2 − a21X1 − a23X3 − · · · − a2nXn

a22... ...

Xn =Cn − an1X1 − an2X3 − · · · − an−1,nXn=1

ann


Jacobi Iteration: Example

◦ Note: New values of variables are not used until a new iteraction isbegun.


Jacobi Iteration: Algorithm


Jacobi Iteration: ApplicationFinite Difference Solution to a PDE


Gauss-Seidel Iteration

� Modification of the Jacobi iteration

� Use the most recent estimates of Xis, instead of waiting for compu-tation of all Xis.

� Results in faster convergence


Gauss-Seidel Iteration: Example2x + y = 6, x = −1

2y + 3

x + 2y = 6, y = −12x + 3

Iterations proceed as follows (assume x(1) = y(1) = 1/2

x(2) = −1

2y(1) + 3 = 11/4, y(2) = −1

2x(2) + 3 = 13/8


Gauss-Seidel Method: Algorithm


Gauss-Seidel Method: Example


Successive Over Relaxation(SOR)

� Goal: To further accelerate Gauss-Seidel iterations using an addi-tional parameter, ω.

� Use a weighted combination of the previous and current updates ofx.

� 0 < ω < 1 (successive underrelaxation), 1 < ω < 2 (SOR), ω = 1 isGauss-Seidel.


Successive Over Relaxation(SOR)

Consider a11 a12 a13 = b1a21 a22 a23 = b2a31 a32 a33 = b3

SOR Equations are

x(new)1 = (1− ω)x

(old)1 +

ω

a11(b1 − a12x

(old)2 − a13x

(old)3 )

x(new)2 = (1− ω)x

(old)2 +

ω

a22(b2 − a21x

(new)1 − a23x

(old)3 )

x(new)3 = (1− ω)x

(old)3 +

ω

a33(b3 − a31x

(new)1 − a32x

(new)2 )


Successive Over Relaxation(SOR):Algorithm


Successive Over Relaxation(SOR):Example


Iterative Methods: Analysis

Consider a matrix A decomposed into E and F ,

A = E + F

(E + F)x = b

Ex = −Fx + b

x = −E−1Fx + E−1b

Or, iteratively,x(k+1) = −E−1Fx(k) + E−1b


Analysis: Jacobi Iteration

In Jacobi’s iteration, the diagonal terms are isolated

A = L + D + U

Ax = b

(L + D + U)x = b

Dx = (−L−U)x + b

x = D−1(−L−U)x + D−1b

Iteratively,

x(k+1) = D−1(−L−U)x(k) + D−1b

x = Cx + d


Analysis: Error

Error term across successive iterations as

e(k+1) = −E−1Fe(k)

For convergence,

limk→∞

e(k) = 0

� If e(k) − e(k+1) is positive, the solution is converging

� Use the norm of e to determine its magnitude (2-norm,∞ norm)


Jacobi Iteration: Convergence� Sufficient Condition: Matrix A be strictly diagonally dominant.

� Necessary and Sufficient Condition: Magnitude of largest eigenvalue of iteration matrix C be larger than 1.

� Order of equations in Jacobi iteration matters!


Jacobi Iteration: Computational Complexity

� Each iteration involves one matrix-vector multiplication, (n−1)2 mul-tiplies

� Total computation can be significantly less than O(n3), required ofGE.

� Particularly appropriate for parallel computation.


Gauss Seidel Iteration: Analysis

A can be decomposed as follows:

A = L + D + U

For Gauss-Seidel iteration,

Ax = b

(D + L)x = −Ux + b

x = (D + L)−1

(−U)x + (D + L)−1

b

x = Tx + c


Gauss Seidel Iteration: Convergence

� Form of the matrix and eigen values determine convergence,but notconvenient for analysis.

� Frequently, matrix is real, symmetric with positive diagonal elements(PDE systems solutions) – convergence guaranteed if eigen valuesare real and positive.

� if A is positive definite, convergence is guaranteed for any initialvector, x(0).

� When iteration matrix is non-negative, Jacobi and Gauss-Seidel ei-ther converge or diverge; Faster convergence using Gauss-Seidel.

� Computational Effort: Roughly half the number of iterations of Ja-cobi.


Successive Overrelaxation: Analysis

Similar to Gauss-Seidel, multiply by relaxation parameter, ω

ω(D + L)x = −ωUx + ωb

Multiply both sides by (1− ω)Dx,

(D− ωL)x = ((1− ω)D− ωU)x + ωb

x = (D− ωL)−1

((1− ω)D− ωU)x + ω(D− ωL)−1

b

⇒ SOR iteration matrix C = (D− ωL)−1

((1− ω)D− ωU).


Successive Overrelaxation: Analysis

� SOR iteration matrix C = (D− ωL)−1

((1− ω)D− ωU).

� Can be shown that |C| = (1− ω)n

� Determinant of C is product of its eigen values.

� Outside of the interval 0 ≤ ω ≤ 2, at least one eigen value is greaterthan 1.

� Choose ω to be between 0 and 2.

� SOR method is designed to reduce the residual, r = b − Ax(k)more rapidly than Gauss-Seidel method.


Successive Overrelaxation: Positive DefiniteMatrices

� SOR method is guaranteed to converge for any positive definite ma-trix.

� Positive Definite Matrix: Following apply:

⇒ xTAx > 0

⇒ All eigen values are positive⇒ All pivots of A are positive


Method of Determinants

Consider the linear systema11 a12 · · · a1n

a21 a22 · · · a2n... ... ... ...

an1 an2 · · · ann

X1X2...

Xn

=

C1C2...

Cn

We can use Cramer’s rule to determine the Xis, as

Xi =|Ai||A|

where |A| is the determinant of A and |Ai| is the determinant of matrixA, whose ith column is replaced by the C vector.


Method of Determinants

For example,

X3 =

∣∣∣∣∣∣∣∣a11 a12 C1 · · · a1n

a21 a22 C2 · · · a2n... ... ... ... ...

an1 an2 Cn · · · ann

∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣a11 a12 a13 · · · a1n

a21 a22 a23 · · · a2n... ... ... ... ...

an1 an2 an3 · · · ann

∣∣∣∣∣∣∣∣ITCS 4133/5133: Intro. to Numerical Methods 30 Iterative Methods

Solving Linear Systems:Iterative Methods · (PDE systems solutions) – convergence guaranteed if...

Documents

Transcript of Solving Linear Systems:Iterative Methods · (PDE systems solutions) – convergence guaranteed if...