Krylov Subspace Methods for Linear Systems and Matrix ...simoncin/siamala.pdf · Convergence...

0.5setgray0

0.5setgray1

Krylov Subspace Methods

for Linear Systems and Matrix Equations

V. Simoncini

Dipartimento di Matematica

Universita di [email protected]

http://www.dm.unibo.it/˜simoncin

Krylov subspace methods – p. 1

Projection methods for large-scale problems

Given the system of n equations

F(x) = 0 x ∈ Rn

- Construct approximation space Km (m = dim(Km) )

- Find x ∈ Km such that x ≈ x

? Projection onto a much smaller space m ¿ n

Approximation process:

residual: r := F(x)

Construct (left) space Lm of dimension m and impose

r ⊥ Lm

Projection methods for large-scale problems

Given the system of n equations

F(x) = 0 x ∈ Rn

- Construct approximation space Km (m = dim(Km) )

- Find x ∈ Km such that x ≈ x

? Projection onto a much smaller space m ¿ n

Approximation process:

residual: r := F(x)

Construct (left) space Lm of dimension m and impose

r ⊥ Lm

Challenges

How to select Km

How to derive good Lm so that it also carries other good properties

General idea:

Construct sequence of approximation spaces Km ⊂ Km+1 such that

xm ∈ Km and xm → x as m → ∞

(in some sense)

Analogously, Lm ⊂ Lm+1

Challenges

How to select Km

General idea:

(in some sense)

Challenges

How to select Km

General idea:

(in some sense)

More specific problems

F(x) = 0

A ∈ Rn×n

Ax− b = 0 (ϕk(A)x− b = 0) where ϕk polynomial of degree k

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

Evaluation of Transfer and other matrix functions

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

F(x) = 0

A ∈ Rn×n

Ax − λMx = 0 (ϕk(λ, A0, A1, . . . , Ak)x = 0)

AX + XAT + Q = 0

Choosing Lm

Focus on linear system:

Ax = b, A nonsing.

xm ∈ Km:

? Lm = Km If A symmetric positive definite,

rm = b − Axm ⊥ Lm ⇔ ‖x − xm‖A = minex∈Km

? Lm = AKm

rm = b − Axm ⊥ Lm ⇔ ‖rm‖2 = minex∈Km

“Optimal” properties hold for any choice of Km Eiermann & Ernst A.N. ’01

Typically: Lm = Km ⇒ FOM, CG Lm = AKm ⇒ GMRES, MINRES

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Choosing Lm

Ax = b, A nonsing.

xm ∈ Km:

? Lm = AKm

Relaxing optimality in Lm. Truncation

Example: A is non-normal, but “nice” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

(-): r ⊥ Lm (- -): r ⊥loc Lm k gives “locality”

Example: A is non-normal, but “nice” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

(-): r ⊥ Lm (- -): r ⊥loc Lm k gives “locality”

Example: A is non-normal, more “nasty” Lm = AKm

0 10 20 30 40 50 60 70 8010−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

space dimension

k = ∞

k = 30

k = 20

k = 10

(-): r ⊥ Lm (- -): r ⊥loc Lm Simoncini &Szyld, Num.Math.’05

Choosing Km

The “classical” approach

Km := Km(A, v) = span{v, Av, . . . , Am−1v}

(e.g., v = b in linear systems)

x ∈ Km, x = pm−1(A)v

Nice Properties:

For m sufficiently large, Km(A, v) invariant for A

Convergence (analysis) in terms of spectral properties of A

Variants of “basic” methods by acting on polynomial pm−1

This choice of Km is good, but no longer sufficiently good

Choosing Km

Nice Properties:

Choosing Km

Nice Properties:

Getting greedy

Km = Km−k(A, v) ∪ Sk New question: Sk?

? Faster approximation when spectral a-priori knowledge is available(even for A Hermitian)

? Main motivating problem:

Large m may be required for accurate approximation xm

Computational/Memory costs increase nonlinearly with m

(A non-normal)

Getting greedy

(A non-normal)

Getting greedy

(A non-normal)

Intermezzo. Restarting procedure

(A non-normal)

Restarting procedure: x0 initial approx, r0 = b − Ax0

Km(A, r0) → x(1)m , r(1)

Km(A, r(1)m ) → x(2)

m , r(2)m

... →...

Warning: Larger m not always implies faster convergence(Embree, Ernst, ...)

Restarted Methods

Convergence strongly depends on choice of m ...

0 100 200 300 400 500 600 700 80010−10

10−8

10−6

10−4

10−2

GMRES(15)

Number of Matrix−vector multiplies

lative r

GMRES(20)

Restarted Methods

Convergence strongly depends on choice of m ... true?

0 100 200 300 400 500 600 700 80010−10

10−8

10−6

10−4

10−2

GMRES(15)

FOM(15)

Global subspace generated

lative r

GMRES(20)

Restarted Methods

Switch to FOM residual vector only at the very first restart

0 50 100 150 200 250 300 350 400 45010−10

10−8

10−6

10−4

10−2

Number of Matrix−vector multiplies

lative

GMRES(15)

GMRES(15)+FOM

Pictures from Simoncini, SIMAX 2000. Intermezzo ends.

Augmented projection spaces

Km = Km−k(A, v) ∪ Sk

Sk from spectral information of A

Km−k(A, v) ∪ Sk = span{v, Av, . . . , Am−k−1v, y1, y2, . . . , yk}

y1, . . . , yk approximate eigenvectors of A associated to cluster

(Baglama, Calvetti, Erhel, Morgan, Nabben, Reichel, Saad, Sorensen, Vuik ...)

Sk important space from previous restartsRanking based on “error” or “effectiveness” of the past spaces

(De Sturler, Baker, Jessup, Manteuffel, ...)

Augmented projection spaces

Km = Km−k(A, v) ∪ Sk

Sk from spectral information of A

Km−k(A, v) ∪ Sk = span{v, Av, . . . , Am−k−1v, y1, y2, . . . , yk}

y1, . . . , yk approximate eigenvectors of A associated to cluster

(Baglama, Calvetti, Erhel, Morgan, Nabben, Reichel, Saad, Sorensen, Vuik ...)

Sk important space from previous restartsRanking based on “error” or “effectiveness” of the past spaces

(De Sturler, Baker, Jessup, Manteuffel, ...)

Augmented method. Spectral Information available.Structural Dynamics problem: (AB−1 + σI)x = b

A,B complex sym. n = 86, 000. Solve for σ in a wide interval

Note: at each iteration solve system with matrix B

B = B(κ) κ related to artificial stiff springs at ground boundaries

B numerically singular as κ → 0

Augmented method. Spectral Information available.Structural Dynamics problem: (AB−1 + σI)x = b

A,B complex sym. n = 86, 000. Solve for σ in a wide interval

Note: at each iteration solve system with matrix B

B = B(κ) κ related to artificial spring stiffness at ground boundaries

B numerically singular as κ → 0

Fill-in p=5 Fill-in p=15

E. Time # its E. Time # its

[s] (outer/ avg. inner) [s] (outer/ avg. inner)

κ = 0 ACG 14066 296/38 12790 281/33

κ = 100 14072 117/120 13739 121/102

κ = 1000 8694 88/96 8724 89/83

κ = 1000 unrealistic Perotti & Simoncini, ’02

ACG: Augmented CG, Saad & Yeung & Ehrel & Guyomarc’h, 2000

Example of Augmented method. Error SpaceTricky way to enhance approximation space

A from L(u) = −1000∆u + 2e4(x2+y2)ux − 2e4(x2+y2)uy n = 40 000

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

Iteration index

tive r

ual norm

GMRES(50)

GMRES(150)

Example of Augmented method. Error SpaceTricky way to enhance approximation space (E. De Sturler)

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

Iteration index

tive r

ual norm

GMRES(50)

GMRES(150)

GCROT(7,23,23,4,1,0)

Code: courtesy of Oliver Ernst see also Baker, Jessup, Manteuffel ’05

Changing Km

Modify Km instead of augmenting it!

Increase flexibility

Cope with “involved” operators

Changing Km. Flexible methods

Original problem

AP−1x = b P preconditioner

Km(AP−1, b) = span{b, AP−1b, . . . , (AP−1)m−1b}

at each iteration i: zi = P−1vi

Flexible variant: Saad, ’93

Iteration i: zi = P−1vi ⇒ zi = P−1i vi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(AP−1, b)

Changing Km. Flexible methods

Original problem

AP−1x = b P preconditioner

Km(AP−1, b) = span{b, AP−1b, . . . , (AP−1)m−1b}

at each iteration i: zi = P−1vi

Flexible variant: Saad, ’93

Iteration i: zi = P−1vi ⇒ zi = P−1i vi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(AP−1, b)

Flexible methods. An exampleFlexible method may be used as a Truncated/Augmented method.

z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

A from L(u) = −∆u + 1000xux n = 900

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

F/G(20) FT/G(16−2) FT/G(12−4)

GM(20)

number of matrix−vector multiplies

tive r

Simoncini & Szyld, SINUM ’03

Flexible methods. An exampleFlexible method may be used as a Truncated/Augmented method.

z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

A from L(u) = −∆u + 1000xux n = 900

0 200 400 600 800 1000 120010−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

F/G(20) FT/G(16−2) FT/G(12−4)

GM(20)

lative r

Flexible methods. A second exampleFlexible method may be used as a Truncated/Augmented method.z = P−1v ⇔ z ≈ A−1v span{b, z1, z2, . . . , zm−1}

0 500 1000 1500 2000 250010−7

10−6

10−5

10−4

10−3

10−2

10−1

lative r

GM(30)

F/G(30)FT/G(10−10)

FT/G(20−5)

Changing Km. Inexact methods.

Original problem

Ax = b A Possibly not available exactly

Km(A, b) = span{b, Ab, . . . , Am−1b}

Inexact (relaxed) variant:

Iteration i: zi = Avi ⇒ zi = Avi + fi

xm ∈ span{b, z1, z2, . . . , zm−1} 6= Km(A, b)

♣ “Worse” than for Flexible methods:

rm = b − Axm not available!

Available: rm computable residual

Original problem

Km(A, b) = span{b, Ab, . . . , Am−1b}

Original problem

Km(A, b) = span{b, Ab, . . . , Am−1b}

Inexact Methods. An example

if ‖fi‖ = O

‖ri−1‖

)∀i ⇒ ‖b − Axm‖ ≤ ε for m large enough

BT S−1B︸︷︷︸A

At each it. i solve:

Swi = Bvi

‖Swi − Bvi‖ ≤ εinner(‖fi‖)

δm = ‖rm − erm‖

0 20 40 60 80 100 12010−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

number of iterations

magnitude

εinner

Simoncini & Szyld, SISC ’03

Inexact Methods. An example

if ‖fi‖ = O

‖ri−1‖

)∀i ⇒ ‖b − Axm‖ ≤ ε for m large enough

BT S−1B︸︷︷︸A

At each it. i solve:

Swi = Bvi

‖Swi − Bvi‖ ≤ εinner(‖fi‖)

δm = ‖rm − erm‖

0 20 40 60 80 100 12010−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

number of iterations

magnitude

εinner

Simoncini & Szyld, SISC ’03

A different application. The Lyapunov equation

AX + XAT + Q = 0

with A dissipative, Q = BBT of low rank X ≈ X low rank

Standard Krylov approach: X ∈ Km(A, B) and

R = AX + XAT + Q ⊥ Lm = Km(A, B)

X = VmYmV Tm for some Ym Range(Vm) = Km(A, B) Saad, ’90

New “Enhanced” approach: X ∈ Kk(A, B) ∪ Kk(A−1, B) and

R = AX + XAT + Q ⊥ Lm = Kk(A, B) ∪ Kk(A−1, B)

? Very competitive w.r.to Cyclic ADI methodSimoncini, tr. 2006

AX + XAT + Q = 0

An example. Time-invariant linear system

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10zxz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Standard Krylov

approximation space dim.: 146 (Standard Krylov) 112 (Enhanced Krylov)

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10zxz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Standard Krylov

x′ = xxx + xyy + xzz − 10xxx − 1000yxy − 10xz + b(x, y)u(t)

A matrix 183 × 183

0 5 10 15 20 2510−11

10−10

10−9

10−8

10−7

10−6

10−5

10−4

10−3

10−2

CPU Time

Enhanced Krylov

Enhanced Krylov(reord)

Standard Krylov

Conclusions and Pointers

Projection is a versatile tool

Lots of room for improvements on hard problems

Survey

“ Recent computational developments in Krylov Subspace Methodsfor linear systems”

with Daniel Szyld, Temple University

To appear in J. Numerical Linear Algebra w/Appl. (352 refs)

This and other papers at

Conclusions and Pointers

Projection is a versatile tool

Lots of room for improvements on hard problems

Survey

“ Recent computational developments in Krylov Subspace Methodsfor linear systems”

with Daniel Szyld, Temple University

To appear in J. Numerical Linear Algebra w/Appl. (352 refs)

This and other papers at

Structural Dynamics Problem

Mq + Cq + Kq = f(t) n d.o.f.

Linearization under small deformations

Direct frequency analysis for damping modeling(as opposed to modal or time analysis)? ideal for mechanical properties depending on frequency- influence of deformation velocity on materials- presence of hysteretic damping, ...

In the frequency domain: (a = −(2πf)2q(f)):

−(2πf)2K∗ +

i(2πf)CV + M

)a = b

C = CV + 12πf

CH , ⇒ K∗ := K + iCH

Concrete Slab ElementsGround Elements

SIDE VIEW

PLAN VIEW

Concrete Slab

Elements

Ground

Elements

Model Problem: 3D Soil-Structure interaction

? viscous damping at the boundary ⇒ K∗ singular

⇒ FE discretization, K∗ almost singular

Algebraic Linearization:(σ2K∗ + σiCV − M

)x = b, x = x(σ)

equivalent to

[iCV −M

−M 0

︸︷︷︸A

[K∗ 0

︸︷︷︸B

with y = σx

Krylov Subspace Methods for Linear Systems and Matrix ...simoncin/siamala.pdf · Convergence...

Documents

Transcript of Krylov Subspace Methods for Linear Systems and Matrix ...simoncin/siamala.pdf · Convergence...