Computational Linear Algebra - cie.bv.tum.demundani/complinalg/part03.pdf · Computational Linear...

Technische Universität München

Computational Linear Algebra

PD Dr. rer. nat. habil. Ralf-Peter MundaniComputation in Engineering / BGU

Scientific Computing in Computer Science / INF

Winter Term 2018/19

PD Dr. Ralf-Peter Mundani – Computational Linear Algebra – Winter Term 2018/19 2


Part 3: Direct Methods



overview

definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition



Definitions

direct methods algorithms for exactly solving linear systems (avoiding round-off errors)

within finite amount of steps nowadays seldom used for solving huge linear systems (due to their large

complexities) nevertheless, frequently used (in incomplete form) as preconditioners

within iterative methods

typical methods GAUSSIAN elimination CHOLESKY decomposition QR decomposition

GRAM-SCHMIDT method GIVENS method …



overview




GAUSSIAN Elimination

basic concept GAUSSIAN elimination (GE) successively transforms a linear system Ax = b

into an equivalent system

LRx = b

with upper triangular matrix R and lower triangular matrix L that can be solved via simple forward / backward substitution

Definition 2.1 The decomposition of a matrix A ∈ into a product

A = LR

consisting of a lower triangular matrix L ∈ and an upper triangular matrix R ∈ is called LR decomposition. In literature this is also referred to as LU (lower, upper) decomposition.




basic concept (cont’d) we denote

as permutation matrix that emerges from the identity matrix I via transposition of j-th and k-th row (j ≥ k) for k = j follows Pkj = I

(2.1.1)

← k-th row

← j-th row

Pkj =




basic concept (cont’d) furthermore, the lower triangular matrix L to be represented via

multiplicative combination of matrices

those matrices differ at most in one column from identity matrix with (2.1.1) and (2.1.2) we are able to formulate the essential part of GE

Lk = (2.1.2)




algorithm: GAUSSIAN elimination (LR decomposition)

with A(n) we get an upper triangular matrix R that can be used for a simple solution of the system

question: where to get matrix L from?

A(1) := A

for k = 1, ..., n−1

choose from k-th column of A(k) some arbitrary element ≠ 0 with j ≥ k

define Pkj with above j and k according to (2.1.1)

Ã(k) := PkjA(k)

define Lk according to (2.1.2) with , i = k+1, ..., n

A(k+1) := LkÃ(k)




existence and uniqueness of LR decomposition simple examples such as

A =

show not each regular matrix necessarily exhibits an LR decomposition

let A ∈ be regular, then A exhibits an LR decomposition if and only if

det A[k] ≠ 0 ∀k = 1, ..., n

with

A[k] := for k ∈ {1, ..., n}

being the principal k×k submatrix of A and det A[k] the principal determinant of A proof is lengthy




algorithm: GAUSSIAN elimination w/o pivoting

for k = 1, ..., n−1for i = k+1, ..., n

aik := aik / akkfor j = k+1, ..., n

aij := aij − aikakjfor k = 2, ..., n

for i = 1, ..., k−1bk := bk − akibi

for k = n, ..., 1for i = k+1, ..., n

bk := bk − akixixk := bk / akk

LR decompositionA := LR

backward substitutionx := R−1b

forward substitutionb := L−1b




complexity only expensive multiplications and divisions to be considered

for k = 1, ..., n−1for i = k+1, ..., n

aik := aik / akkfor j = k+1, ..., n

aij := aij − aikakjfor k = 2, ..., n

for i = 1, ..., k−1bk := bk − akibi

for k = n, ..., 1for i = k+1, ..., n

bk := bk − akixixk := bk / akk n-times

(n−k)-times (inner loop) for k = 1, ..., n−1

(n−k)2-times (inner loops) for k = 1, ..., n−1

(n−k)-times (inner loop) for k = 1, ..., n

(k−1)-times (inner loop) for k = 2, ..., n




complexity (cont’d) only expensive multiplications and divisions to be considered

hence, #divisions + #multiplications = such that the total complexity of GAUSSIAN elimination can be estimated as Ο(n3)

on a standard computer (3 GHz) we need for n = 104 approx. 5 minutes

#divisions =

#multiplications =

=




error analysis let ε 1, such that (according to machine precision)

resp.

applies

consider the following linear system Ax = b with

and its exact solution




error analysis (cont’d) with GAUSSIAN elimination we get

and due to present computational accuracy

substituting this into first equation of (1) yields

hence x1 = 0 follows

(1)




error analysis (cont’d) a previous row interchange yields the following linear system

hence, with GAUSSIAN elimination follows

and we get x2 = 0.5 and x1 = 1 − x2 = 0.5 as (correct) solution

permutation of rows and columns not only makes sense in case such strategies are called pivoting




pivoting we consider three different types of pivoting: column pivoting

define Pkj according to (2.1.1) with

j = index

and consider for

A(k)x = b

the equivalent linear system

PkjA(k)x = Pkjb




pivoting (cont’d) we consider three different types of pivoting: row pivoting

define Pkj according to (2.1.1) with

j = index

and consider the linear system

A(k) Pkjy = b

x = Pkjy




pivoting (cont’d) we consider three different types of pivoting: total pivoting

define Pkj1, Pkj2 according to (2.1.1) with

j1 = index

j2 = index

and consider the linear system

Pkj1A(k) Pkj2y = Pkj1b

x = Pkj2y



overview




CHOLESKY Decomposition

basic concept complexity for LR decomposition of symmetric positive definite (SPD)

matrices during GAUSSIAN elimination can be further reduced


A = LLT

with a lower triangular matrix L ∈ is called CHOLESKY decomposition.

for each SPD matrix A ∈ exists exactly one lower triangular matrixL ∈ with lii > 0, i = 1, ..., n, such that A = LLT applies




basic concept (cont’d) let’s consider a column-wise computation of matrix coefficients we assume all lij for i = 1, ..., n and j ≤ k−1 are known hence, from

follows the relation

thus lkk can be computed according to

(2.2.1)




basic concept (cont’d) from

for i = k+1, ..., n

follows a rule for computing elements of k-th column (below diagonal) via

for i = k+1, ..., n (2.2.2)

with (2.2.1) and (2.2.2) we are able to formulate the algorithm for the CHOLESKY decomposition




algorithm

for k = n, ..., 1for i = k+1, ..., n

bk := bk − aikxixk := bk / akk

CHOLESKY decompositionA := LLT

backward substitutionx := L−Tb

for k = 1, ..., nfor i = 1, ..., k−1

bk := bk − akibibk := bk / akk

forward substitutionb := L−1b

for k = 1, ..., nfor j = 1, ..., k−1

akk := akk − akjakjakk := (2

.2.1

)

for i = k+1, ..., nfor j = 1, ..., k−1

aik := aik − aijakjaik := aik / akk

(2.2

.2)




complexity solely decomposition (as most complex part) to be considered only expensive multiplications, divisions, and roots

for k = 1, ..., nfor j = 1, ..., k−1

akk := akk − akjakjakk :=for i = k+1, ..., n

for j = 1, ..., k−1aik := aik − aijakj

aik := aik / akk (n−k)-times (inner loop) for k = 1, ..., n

(n−k)⋅(k−1)-times (inner loops) for k = 1, ..., n

n-times

(k−1)-times (inner loop) for k = 1, ..., n




complexity (cont’d) #multiplications

hence, for large n CHOLESKY decomposition needs approximately only half of the expensive operations of GAUSSIAN elimination

0 0

+ #divisions + #roots



overview




QR Decomposition

basic concept essential foundation for GMRES ( iterative method) and many methods

for solving eigenvalue problems or linear regressions due to Q∗ = Q−1, the linear system Ax = b is easily to be solved via

Ax = b ↔ QRx = b ↔ Rx = Q∗b

the three most well-known methods for this decomposition are GIVENS

GRAM-SCHMIDT or HOUSEHOLDER (both not to be considered here)


A = QR

with a unitary matrix Q ∈ and an upper triangular matrix R ∈ is called QR decomposition.



QR Decomposition

GIVENS method let’s assume the following matrix (confined to the case A ∈ )

A =

← j-th row

↑ i-th column



QR Decomposition

GIVENS method (cont’d) idea: successively eliminate elements below main diagonal starting from first column, sub-diagonal elements of each column become

nullified in ascending order via orthogonal rotation matrices

for previous matrix A applies

akl = 0 ∀ l ∈ {1, ..., i−1} with l < k ∈ {1, ..., n},

ai+1,i = ... = aj−1,i = 0

and aji ≠ 0

in order to nullify aji, we look for an orthogonal matrix Gji ∈

(2.3.1)

(2.3.2)



QR Decomposition

GIVENS method (cont’d) let

be an orthogonal matrix

Gji =



QR Decomposition

GIVENS method (cont’d) such that for

Ã = GjiA

in addition to

ãkl = 0 ∀ l ∈ {1, ..., i−1} with l < k ∈ {1, ..., n},

and

ãi+1,i = ... = ãj−1,i = 0

also

ãji = 0

applies

(2.3.3)

(2.3.4)

(2.3.5)



QR Decomposition

GIVENS method (cont’d) at first, Ã and A solely differ in i-th and j-th row, and for l = 1, ..., n

ãil = giiail + gijajl

ãjl = gjiail + gjjajl

applies – with (2.3.1) follows ail = ajl = 0 for l < i < j, thus

ãil = ãjl = 0 for l = 1, ..., i−1

and hence the requirements (2.3.3) and (2.3.4) are fulfilled

well-defined via aji ≠ 0 we set

gij = −gji = and gii = gjj =



QR Decomposition

GIVENS method (cont’d) thus, Gji represents an orthogonal rotation matrix with angle α = arccos gii

and the following

applies

defining Gji = I in case of a matrix A that satisfies (2.3.1) and (2.3.2) and furthermore implies aji = 0, then with

Q := Gji := Gn,n−1 ⋅ ... ⋅ G3,2 ⋅ Gn,1 ⋅ ... ⋅ G3,1 ⋅ G2,1

we get an orthogonal matrix for which

R = QA

yields an upper triangular matrix – with Q = QT follows A = QR~

~

~



QR Decomposition

algorithm

ajk := −saik + cajk

for i = 1, ..., n−1for j = i+1, ..., n

aji = 0

k = i

Y

Y

N

N

t := 1 /

s := tajic := taiifor k = i, ..., n+1

t := caik + sajk

aik := taji := 0

QR

deco

mpo

sitio

nA

:=Q

R

for i = n, ..., 1for j = i+1, ..., n

ai,n+1 := ai,n+1 − aijxjxi := ai,n+1 / aii

back substitutionx := R−1b

By using GIVENS method, thereis no need to explicitly store theorthogonal matrix. Here, A willbe extended by the right-handside b according to an+1 = b.



QR Decomposition

complexity

ajk := −saik + cajk

for i = 1, ..., n−1for j = i+1, ..., n

aji = 0

k = i

Y

Y

N

N

t := 1 /

s := tajic := taiifor k = i, ..., n+1

t := caik + sajk

aik := taji := 0

(n−i)-times (inner loop) for i = 1, ..., n−1

(n−i)-times (inner loop) for i = 1, ..., n−1

2(n−i+1)(n−i)-times (inner loops) for i = 1, ..., n−1

2(n−i+1)(n−i)-times (inner loops) for i = 1, ..., n−1

(n−i)-times (inner loop) for i = 1, ..., n−1 (n−i)-times (inner loop) for i = 1, ..., n−1 (n−i)-times (inner loop) for i = 1, ..., n−1

solely decomposition (as mostcomplex part) to be consideredw/o right-hand side b

only expensive multiplications,divisions, and roots



QR Decomposition

complexity (cont’d) #multiplications

hence, the complexity is approximately four times larger than GAUSSIANelimination

+ #divisions + #roots



overview


Computational Linear Algebra - cie.bv.tum.demundani/complinalg/part03.pdf · Computational Linear...

Documents

Transcript of Computational Linear Algebra - cie.bv.tum.demundani/complinalg/part03.pdf · Computational Linear...