Numerical Linear Algebra · 2020. 11. 19. · Numerical Linear Algebra Victor Eijkhout Fall 2020....

Numerical Linear Algebra

Victor Eijkhout

Spring 2021

Justification

Many algorithms are based in linear algebra, including somenon-obvious ones such as graph algorithms. This session will mostlydiscuss aspects of solving linear systems, focusing on those that havecomputational ramifications.

SDS 374c/394c — Spring 2021— 2

Linear algebra

• Mathematical aspects: mostly linear system solving• Practical aspects: even simple operations are hard

– Dense matrix-vector product: scalability aspects– Sparse matrix-vector: implementation

Let’s start with the math. . .

SDS 374c/394c — Spring 2021— 3

Two approaches to linear system solvingSolve Ax = b

Direct methods:

• Deterministic

• Exact up to machine precision

• Expensive (in time and space)

Iterative methods:

• Only approximate

• Cheaper in space and (possibly) time

• Convergence not guaranteed

SDS 374c/394c — Spring 2021— 4

Really bad example of direct method

Cramer’s rulewrite |A| for determinant, then

xi =

∣∣∣∣∣∣∣∣∣a11 a12 . . . a1i−1 b1 a1i+1 . . . a1na21 . . . b2 . . . a2n

......

...an1 . . . bn . . . ann

∣∣∣∣∣∣∣∣∣/|A|Time complexity O(n!)

SDS 374c/394c — Spring 2021— 5

Not a good method either

Ax = b

• Compute explictly A−1,

• then x ← A−1b.• Numerical stability issues.

• Amount of work?

SDS 374c/394c — Spring 2021— 6

A close look linear systemsolving: direct methods

SDS 374c/394c — Spring 2021— 7

Gaussian elimination

Example 6 −2 212 −8 63 −13 3

x = 1626−19

6 −2 2 | 1612 −8 6 | 26

3 −13 3 | −19

−→6 −2 2 | 160 −4 2 | −6

0 −12 2 | −27

−→6 −2 2 | 160 −4 2 | −6

0 0 −4 | −9

Solve x3, then x2, then x1

6,−4,−4 are the ‘pivots’

SDS 374c/394c — Spring 2021— 8

Gaussian elimination, step by step

〈LU factorization〉:for k = 1,n−1:〈eliminate values in column k〉

〈eliminate values in column k〉:for i = k + 1 to n:〈compute multiplier for row i〉〈update row i〉

〈compute multiplier for row i〉aik ← aik/akk

〈update row i〉:for j = k + 1 to n:

aij ← aij −aik ∗akj

SDS 374c/394c — Spring 2021— 9

Gaussian elimination, all together

〈LU factorization〉:for k = 1,n−1:

for i = k + 1 to n:aik ← aik/akkfor j = k + 1 to n:

aij ← aij −aik ∗akj

Amount of work:

n−1

∑k=1

∑i,j>k

1 =n−1

∑k

(n− k)2 ≈∑k

k2 = O(n3)

SDS 374c/394c — Spring 2021— 10

Pivoting

If a pivot is zero, exchange that row and another.(there is always a row with a nonzero pivot if the matrix is nonsingular)best choice is the largest possible pivotin fact, that’s a good choice even if the pivot is not zero:partial pivoting(full pivoting would be row and column exchanges)

SDS 374c/394c — Spring 2021— 11

Roundoff controlConsider (

ε 11 1

)x =

(1 + ε

2

)with solution x = (1,1)t

Ordinary elimination:(ε 10 1− 1ε

)x =

(1 + ε

2− 1+εε

)=

(1 + ε1− 1ε

).

We can now solve x2 and from it x1:{x2 = (1− ε−1)/(1− ε−1) = 1x1 = ε−1(1 + ε− x2) = 1.

SDS 374c/394c — Spring 2021— 12

Roundoff 2If ε < εmach, then in the rhs 1 + ε→ 1, so the system is:(

ε 11 1

)x =

(12

)The solution (1,1) is still correct!

Eliminating:(ε 10 1− ε−1

)x =

(1

2− ε−1)⇒

(ε 10 −ε−1

)x =

(1−ε−1

)Solving first x2, then x1, we get:{

x2 = ε−1/ε−1 = 1x1 = ε−1(1−1 · x2) = ε−1 ·0 = 0,

so x2 is correct, but x1 is completely wrong.

SDS 374c/394c — Spring 2021— 13

Roundoff 3

Pivot first:(1 1ε 1

)x =

(2

1 + ε

)⇒(

1 10 1− ε

)x =

(2

1− ε

)Now we get, regardless the size of epsilon:

x2 =1− ε1− ε

= 1, x1 = 2− x2 = 1

SDS 374c/394c — Spring 2021— 14

LU factorizationSame example again:

A =

6 −2 212 −8 63 −13 3

2nd row minus 2× first; 3rd row minus 1/2× first;equivalent to

L1Ax = L1b, L1 =

1 0 0−2 1 0−1/2 0 1

(elementary reflector)

SDS 374c/394c — Spring 2021— 15

LU 2

Next step: L2L1Ax = L2L1b with

L2 =

1 0 00 1 00 −3 1

Define U = L2L1A, then A = LU with L = L

−11 L

−12

‘LU factorization’

SDS 374c/394c — Spring 2021— 16

LU 3Observe:

L1 =

1 0 0−2 1 0−1/2 0 1

L−11 = 1 0 02 1 0

1/2 0 1

Likewise

L2 =

1 0 00 1 00 −3 1

L2 =1 0 00 1 0

0 3 1

Even more remarkable:

L−11 L−12 =

1 0 02 1 01/2 3 1

Can be computed in place! (pivoting?)

SDS 374c/394c — Spring 2021— 17

Solve LU systemAx = b −→ LUx = b solve in two steps:Ly = b, and Ux = y

Forward sweep:1 /0`21 1`31 `32 1...

. . .`n1 `n2 · · · 1

y1y2

...yn

=

b1b2

...bn

y1 = b1, y2 = b2− `21y1, . . .

SDS 374c/394c — Spring 2021— 18

Solve LU 2

Backward sweep:u11 u12 . . . u1n

u22 . . . u2n. . .

.../0 unn

x1x2...

xn

=

y1y2...

yn

xn = u−1nn yn, xn−1 = u

−1n−1n−1(yn−1−un−1nxn), . . .

SDS 374c/394c — Spring 2021— 19

Solve LU 2

Backward sweep:u11 u12 . . . u1n

u22 . . . u2n. . .

.../0 unn

x1x2...

xn

=

y1y2...

yn

xn = u

−1nn yn, xn−1 = u

−1n−1n−1(yn−1−un−1nxn), . . .

SDS 374c/394c — Spring 2021— 19

Computational aspects

Compare:

Matrix-vector product:y1...yn

←a11 . . . a1n... ...

an1 . . . ann

x1...

xn

Solving LU system:a11 /0... . . .

an1 . . . ann

x1...

xn

y1...

yn

(and similarly the U matrix)

Compare operation counts. Can you think of other points ofcomparison? (Think modern computers.)

SDS 374c/394c — Spring 2021— 20

Short detour: PartialDifferential Equations

SDS 374c/394c — Spring 2021— 21

Second order PDEs; 1D case{−u′′(x) = f (x) x ∈ [a,b]u(a) = ua, u(b) = ub

Using Taylor series:

u(x +h)+u(x−h) = 2u(x)+u′′(x)h2 +u(4)(x) h4

12+ · · ·

so

u′′(x) =u(x +h)−2u(x)+u(x−h)

h2+O(h2)

Numerical scheme:

−u(x +h)−2u(x)+u(x−h)h2

= f (x ,u(x),u′(x))

SDS 374c/394c — Spring 2021— 22

This leads to linear algebra

−uxx = f →2u(x)−u(x + h)−u(x−h)

h2= f (x ,u(x),u′(x))

Equally spaced points on [0,1]: xk = kh where h = 1/(n + 1), then

−uk+1 + 2uk −uk−1 =−h2 f (xk ,uk ,u′k ) for k = 1, . . . ,n

Written as matrix equation: 2 −1−1 2 −1. . . . . . . . .

u1u2

...

=f1 + u0f2

...

SDS 374c/394c — Spring 2021— 23

Second order PDEs; 2D case

{−uxx (x̄)−uyy (x̄) = f (x̄) x ∈ Ω = [0,1]2

u(x̄) = u0 x̄ ∈ δΩ

Now using central differences in both x and y directions.

SDS 374c/394c — Spring 2021— 24

The stencil view of things

SDS 374c/394c — Spring 2021— 25

Sparse matrix from 2D equation

4 −1 /0 −1 /0−1 4 1 −1

. . .. . .

. . .. . .

. . .. . . −1

. . ./0 −1 4 /0 −1−1 /0 4 −1 −1

−1 −1 4 −1 −1

↑. . . ↑ ↑ ↑ ↑

k−n k−1 k k + 1 −1 k + n−1 −1 4

. . .. . .

The stencil view is often more insightful.

SDS 374c/394c — Spring 2021— 26

Matrix properties

• Very sparse, banded

• Factorization takes less than n2 space, n3 work

• Symmetric (only because 2nd order problem)

• Sign pattern: positive diagonal, nonpositive off-diagonal(true for many second order methods)

• Positive definite (just like the continuous problem)

• Constant diagonals: only because of the constant coefficientdifferential equation

• Factorization: lower complexity than dense, recursion length lessthan N.

SDS 374c/394c — Spring 2021— 27

Sparse matrices

SDS 374c/394c — Spring 2021— 28

Sparse matrix storageMatrix above has many zeros: n2 elements but only O(n) nonzeros.Big waste of space to store this as square array.

Matrix is called ‘sparse’ if there are enough zeros to make specializedstorage feasible.

SDS 374c/394c — Spring 2021— 29

Compressed Row Storage

A =

10 0 0 0 −2 03 9 0 0 0 30 7 8 7 0 03 0 8 7 5 00 8 0 9 9 130 4 0 0 2 −1

. (1)Compressed Row Storage (CRS): store all nonzeros by row, theircolumn indices, pointers to where the columns start (1-basedindexing):

val 10 -2 3 9 3 7 8 7 3 · · · 9 13 4 2 -1col ind 1 5 1 2 6 2 3 4 1 · · · 5 6 2 5 6

row ptr 1 3 6 9 13 17 20 .

SDS 374c/394c — Spring 2021— 30

Sparse matrix-vector operations

• Simplest, and important in many contexts: matrix-vector product.

• Matrix-matrix product does not happen much.

• Gaussian elimination is a complicated story. Out of scope.

• In general: changes to sparse structure are hard!

SDS 374c/394c — Spring 2021— 31

Dense matrix-vector productMost common operation in many cases: matrix-vector product

aptr = 0;for (row=0; row

Sparse matrix-vector productaptr = 0;for (row=0; row

Exercise: sparse coding

What if you need access to both rows and columns at the same time?Implement an algorithm that tests whether a matrix stored in CRSformat is symmetric. Hint: keep an array of pointers, one for each row,that keeps track of how far you have progressed in that row.

SDS 374c/394c — Spring 2021— 34

Fill-in

Remember Gaussian elimination algorithm:

for k = 1,n−1:for i = k + 1 to n:

for j = k + 1 to n:aij ← aij −aik ∗akj/akk

Fill-in: index (i, j) where aij = 0 originally, but gets updated to non-zero.(and so `ij 6= 0 or uij 6= 0.)

Change in the sparsity structure! How do you deal with that?

SDS 374c/394c — Spring 2021— 35

LU of a sparse matrix

2 −1 0 . . .−1 2 −10 −1 2 −1

. . . . . . . . . . . .

⇒

2 −1 0 . . .0 2− 12 −10 −1 2 −1

. . . . . . . . . . . .

How does this continue by induction?

Observations?

SDS 374c/394c — Spring 2021— 36

LU of a sparse matrix

4 −1 0 . . . −1−1 4 −1 0 . . . 0 −1

. . .. . .

. . .. . .

−1 0 . . . 4 −10 −1 0 . . . −1 4 −1

⇒

4 −1 0 . . . −1

4− 14 −1 0 . . . −1/4 −1. . .

. . .. . .

. . .−1/4 . . . 4− 14 −1−1 0 . . . −1 4 −1

SDS 374c/394c — Spring 2021— 37

A little graph theoryGraph is a tuple G = 〈V ,E〉 where V = {v1, . . .vn} for some n, andE ⊂ {(i, j) : 1≤ i, j ≤ n, i 6= j}.

{V = {1,2,3,4,5,6}E = {(1,2),(2,6),(4,3),(4,4),(4,5)}

SDS 374c/394c — Spring 2021— 38

Graphs and matricesFor a graph G = 〈V ,E〉, the adjacency matrix M is defined by

Mij =

{1 (i, j) ∈ E0 otherwise

A dense and a sparse matrix, both with their adjacency graph

SDS 374c/394c — Spring 2021— 39

Fill-inFill-in: index (i, j) where aij = 0 originally, but gets updated to non-zero.

aij ← aij −aik ∗akj/akk

Eliminating a vertex introduces a new edge in the quotient graph

SDS 374c/394c — Spring 2021— 40

LU of sparse matrix, with graph view: 1

Original matrix.

SDS 374c/394c — Spring 2021— 41


Eliminating (2,1) causes fill-in at (2,3).

SDS 374c/394c — Spring 2021— 42


Remaining matrix when step 1 finished.

SDS 374c/394c — Spring 2021— 43


Eliminating (3,2) fills (3,4)

SDS 374c/394c — Spring 2021— 44


After step 2

SDS 374c/394c — Spring 2021— 45

Fill-in is a function of ordering

∗ ∗ · · · ∗∗ ∗ /0...

. . .∗ /0 ∗

After factorization the matrix is dense.Can this be permuted?

SDS 374c/394c — Spring 2021— 46

Exercise: LU of a penta-diagonal matrixConsider the matrix

2 0 −10 2 0 −1−1 0 2 0 −1

−1 0 2 0 −1. . . . . . . . . . . . . . .

Describe the LU factorization of this matrix:

• Convince yourself that there will be no fill-in. Give an inductiveproof of this.

• What does the graph of this matrix look like? (Find a tutorial ongraph theory. What is a name for such a graph?)

• Can you relate this graph to the answer on the question of thefill-in?

SDS 374c/394c — Spring 2021— 47

Exercise: LU of a band matrix

Suppose a matrix A is banded with halfbandwidth p:

aij = 0 if |i− j|> p

Derive how much space an LU factorization of A will take if no pivotingis used. (For bonus points: consider partial pivoting.)

Can you also derive how much space the inverse will take? (Hint: ifA = LU, does that give you an easy formula for the inverse?)

SDS 374c/394c — Spring 2021— 48

Numerical Linear Algebra · 2020. 11. 19. · Numerical Linear Algebra Victor Eijkhout Fall 2020....

Documents

Transcript of Numerical Linear Algebra · 2020. 11. 19. · Numerical Linear Algebra Victor Eijkhout Fall 2020....