Preconditioned Inverse Iteration for Hierarchical Matrices

MAX PLANCK INSTITUTE

FOR DYNAMICS OF COMPLEX

TECHNICAL SYSTEMS

MAGDEBURG

82nd GAMM Annual Scientific ConferenceGraz, April 19, 2011

Preconditioned Inverse Iteration forHierarchical Matrices

Peter Benner and Thomas Mach

Max Planck Institute for Dynamics of Complex Technical SystemsComputational Methods in Systems and Control Theory

Magdeburg

Max Planck Institute Magdeburg Thomas Mach, Preconditioned Inverse Iteration for Hierarchical Matrices 1/28

Hierarchical (H-)Matrices PINVIT Numerical Results

Outline

1 Hierarchical (H-)Matrices

2 PINVIT

3 Numerical Results

H-Matrices [Hackbusch 1998]

Some dense matrices, e.g. BEM or FEM, can be approximated byH-matrices in a data-sparse manner.

hierarchical tree TI block H-tree TI× I

I = {1, 2, 3, 4, 5, 6, 7, 8}

{1, 2, 3, 4} {5, 6, 7, 8}

{1, 2} {3, 4} {5, 6} {7, 8}

{1}{2}{3}{4}{5}{6}{7}{8}

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 56 6 6 67 7 7 78 8 8 8

dense matrices, rank-k-matrices

rank-k-matrix: Ms×t = ABT , A ∈ Rn×k ,B ∈ Rm×k (k � n,m)

Hierarchical matrices

H(TI× I, k) ={M ∈ RI× I

∣∣ rank (Ma×b) ≤ k ∀a× b admissible}

14 118 11

10 31 11

9 16 12

11 1611 8

9 16 11

619 73 3

10 19 11

8 15 8 12

116 515 6

8 11 8 11

5 8 11

126 15

5 6 13

5 8 11

116 15

6110 103

10 3 6

25 10 6

9 11 10

10 1615 9

8 11 10

5110 107 9 7

1125 10

11 811 8

8 15 9

8 15 12 13

1310 7

8 11 9

9 15 1119

2010 7

8 15 11 12

8 1111

1011 8

8 15 8

7 12 13

8 11 8

7 1120

7 11 12

3 10 107 10 6

7 10 7

1011 8

7 1113 7

10 11 61

6 513 6 11

8 15 812

13 6 11

11 236 13

5 6 12

5 6 10

7 10 61

1015 9

9 107 13 51

adaptive rank k(ε)

storage NSt,H(T , k) = O(n log n k(ε))

complexity of approximate arithmetic

MHv O(n log n k(ε))+H,−H O(n log n k(ε)2)

∗H,HLU(·), (·)−1H O(n (log n)2 k(ε)2)

Hierarchical matrices

H(TI× I, k) ={M ∈ RI× I

∣∣ rank (Ma×b) ≤ k ∀a× b admissible}

14 118 11

10 31 11

9 16 12

11 1611 8

9 16 11

619 73 3

10 19 11

8 15 8 12

116 515 6

8 11 8 11

5 8 11

126 15

5 6 13

5 8 11

116 15

6110 103

10 3 6

25 10 6

9 11 10

10 1615 9

8 11 10

5110 107 9 7

1125 10

11 811 8

8 15 9

8 15 12 13

1310 7

8 11 9

9 15 1119

2010 7

8 15 11 12

8 1111

1011 8

8 15 8

7 12 13

8 11 8

7 1120

7 11 12

3 10 107 10 6

7 10 7

1011 8

7 1113 7

10 11 61

6 513 6 11

8 15 812

13 6 11

11 236 13

5 6 12

5 6 10

7 10 61

1015 9

9 107 13 51

adaptive rank k(ε)

storage NSt,H(T , k) = O(n log n k(ε))

complexity of approximate arithmetic

MHv O(n log n k(ε))+H,−H O(n log n k(ε)2)

∗H,HLU(·), (·)−1H O(n (log n)2 k(ε)2)

= A1 A2

Goal: Eigenvalues of symmetric H-Matrices

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λi in O(n (log n)α kβ)?

How to find λi in O(n (log n)α kβ)?

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

Preconditioned Inverse Iteration[Knyazev, Neymeyr, et al.]

Definition

The function

µ(x) = µ(x ,M) =xTMx

is called the Rayleigh quotient.

Minimize the Rayleigh quotient by a gradient method:

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xT x(Mx − xµ(x)) ,

+ preconditioning ⇒ update equation:

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Residual r(x) = Mx − xµ(x).

Preconditioned residual B−1r(x) = B−1 (Mx − xµ(x)).

Definition

The function

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Definition

The function

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Definition

The function

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Definition

The function

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Preconditioned Inverse Iteration[Knyazev, Neymeyr 2009]

xi+1 := xi − B−1 (Mxi − xiµ(xi ))

M ∈ Rn×n symmetric positive definite and

B−1 approximates the inverse of M, so that∥∥I − B−1M∥∥M≤ c < 1,

then Preconditioned INVerse ITeration (PINVIT) converges andthe number of iterations is independent of n.

The residual

ri = Mxi − xiµ(xi )

converges to 0, so that

‖ri‖2 < ε

is a useful termination criterion.

Variants of PINVIT[Neymeyr 2001: A Hierarchy of Precond. Eigens. for Ellipt. Diff. Op.]

Classification by Neymeyr:

PINVIT(1): xi+1 := xi − B−1ri .

PINVIT(2): xi+1 := arg minv∈span{xi ,B−1ri} µ(v).

PINVIT(3): xi+1 := arg minv∈span{xi−1,xi ,B−1ri} µ(v).

PINVIT(n): Analogously.

PINVIT(·,d): Replacing x by a rectangular full rank matrixX ∈ Rn×d one gets the subspace version of PINVIT(·).

Algorithm and Complexity

The number of iterations is independent of matrix size n.

H-PINVIT(1,d)

Input: M ∈ Rn×n, X0 ∈ Rn×d (XT0 X0 = I , e.g. randomly chosen)

Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1H or B−1 = L−TH L−1H

O(n (log n)2 k (c)2)

R := MX0 − X0µ, µ = XT0 MX0

for (i := 1;‖R‖F > ε; i + +) do

Xi := Orthogonalize(Xi−1 − B−1R

O(n (log n) k (c)2)

R := MXi − Xiµ, µ = XTi MXi

O(n (log n) k (c))

The complexity of the algorithm is determined by the H-matrixinversion/Cholesky decomposition: ⇒ O(n (log n)2 k (c)2).

Competitive to MATLAB® eigs.

Expensive.

H-PINVIT(1,d)

Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1H or B−1 = L−TH L−1H O(n (log n)2 k (c)2)R := MX0 − X0µ, µ = XT

for (i := 1;‖R‖F > ε; i + +) do

)O(n (log n) k (c)2)

R := MXi − Xiµ, µ = XTi MXi O(n (log n) k (c))

Competitive to MATLAB eigs.

Expensive.

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)? X

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

Folded Spectrum Method

If i = n − d with d < O(log n),

use subspace version PINVIT(·,d).

0λn λn−1λn−2. . .

But (M − σI) is not positive definite.

If i = n − d with d � log n?

shift with σ near λi .

0λn. . .

λi+1 λi λi−1. . .

If i = n − d with d � log n,

0λn. . .

λi+1 λi λi−1. . .

If i = n − d with d � log n,

0λn. . .

λi+1 λi λi−1. . .

Folded Spectrum Method [Wang, Zunger 1994]

Mσ = (M − σI)2

Mσ is s.p.d., if M is s.p.d. and σ 6= λi .

Assume all eigenvalues of Mσ are simple.

Mv = λv ⇔ Mσv = (M − σI)2 v

= M2v − 2σMv + σ2v

= λ2v − 2σλv + σ2v

= (λ− σ)2v

1 Choose σ.2 Compute

a) Mσ = (M − σI)2 andb) M−1

σ or LLT = Mσ.

3 Use PINVIT to find the smallest eigenpair (µσ, v) of Mσ.4 Compute µ = vTMv/vTv .

(µ, v) is the nearest eigenpair to σ.

H-arithmetic enables us to shift, square and invert M resp.Mσ in linear-polylogarithmic complexity.

If M is sparse, then shifting, squaring and inverting isprohibitive.

σ or LLT = Mσ.

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

How to find λi in O(n (log n)α kβ)? X

Numerical Results

Hlib [Borm, Grasedyck, et al.]

We use the Hlib1.3 (www.hlib.org) for theH-arithmetic operations and some examples out ofthe library for testing the eigenvalue algorithm.

2D Laplace, smallest eigenvalues

2D Laplace over [−1, 1]× [−1, 1], precond.: H-inversion

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 5.6146E-010 0.01

FEM16 256 4.5918E-010 0.02 2.00 106.67

FEM32 1 024 3.7550E-010 0.12 6.17 27.08

FEM64 4 096 3.8009E-010 0.82 6.68 8.06

FEM128 16 384 4.4099E-010 5.84 7.09 5.44

FEM256 65 536 3.9651E-010 34.47 5.91 5.22

FEM512 262 144 3.7877E-010 194.00 5.63 6.51

d = 4, c = 0.2, εeig = 10−4, N(ni ) = ni (log2 ni )CspCid

Time t only H-PINVIT (without H-inversion)

∥∥∥∥∥∥∥∥∥

λ1 − λ1λ2 − λ2λ3 − λ3λ4 − λ4

∥∥∥∥∥∥∥∥∥2

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 5.6146E-010 0.01

FEM16 256 4.5918E-010 0.02 2.00 106.67

FEM32 1 024 3.7550E-010 0.12 6.17 27.08

FEM64 4 096 3.8009E-010 0.82 6.68 8.06

FEM128 16 384 4.4099E-010 5.84 7.09 5.44

FEM256 65 536 3.9651E-010 34.47 5.91 5.22

FEM512 262 144 3.7877E-010 194.00 5.63 6.51

∥∥∥∥∥∥∥∥∥

λ1 − λ1λ2 − λ2λ3 − λ3λ4 − λ4

∥∥∥∥∥∥∥∥∥2

FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

H-PINVIT(1,4) H-inversion

H-PINVIT(3,4) O(N(ni ) log ni )

O(N(ni ))

FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

H-PINVIT(1,4) H-inversion

O(N(ni )) MATLAB eigs

H-PINVIT(1,d)

for (i := 1;‖R‖F > ε; i + +) do

Expensive.

H-PINVIT(1,d)

for (i := 1;‖R‖F > ε; i + +) do

Expensive.

H-PINVIT(1,d)

for (i := 1;‖R‖F > ε; i + +) do

Expensive.

2D Laplace over [−1, 1]× [−1, 1], precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 4.6920E-010 0.01

FEM16 256 4.7963E-010 0.02 2.00 106.67

FEM32 1 024 3.4696E-010 0.08 4.00 27.08

FEM64 4 096 4.6414E-010 0.48 6.00 8.06

FEM128 16 384 3.3206E-010 3.20 6.67 5.44

FEM256 65 536 3.8468E-010 13.90 4.34 5.22

FEM512 262 144 3.1353E-010 62.40 4.49 6.51

Time t only H-PINVIT (without H-Cholesky decomposition)

∥∥∥∥∥∥∥∥∥

λ1 − λ1λ2 − λ2λ3 − λ3λ4 − λ4

∥∥∥∥∥∥∥∥∥2

FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

H-PINVIT(1,4) H-Chol. decomp.

O(N(ni ))

FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

H-PINVIT(1,4) H-Chol. decomp.

O(N(ni )) MATLAB eigs

2D Laplace, vn, FEM64

−1 −0.5 00.5 1−1

·10−2

6·10−2

2D Laplace, vn−1, FEM64

−1 −0.5 00.5 1−1

·10−2

6·10−2

−1 −0.5 00.5 1−1

·10−2

6·10−2

−1 −0.5 00.5 1−1

·10−2

6·10−2

2D Laplace, inner eigenvalues

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 1.7219E-013 <0.01

FEM16 256 7.2388E-012 0.03 106.67

FEM32 1 024 6.0045E-013 0.12 3.57 27.08

FEM64 4 096 1.0915E-012 0.93 7.48 8.06

FEM128 16 384 1.5655E-012 6.08 6.50 5.44

FEM256 65 536 1.1976E-011 52.40 8.62 5.22

2D Laplace, inner eigenvalues

2D Laplace over [−1, 1]× [−1, 1], precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 1.5306E-013 <0.01

FEM16 256 9.8071E-012 0.02 106.67

FEM32 1 024 1.0577E-012 0.05 2.50 27.08

FEM64 4 096 1.5776E-012 0.31 6.20 8.06

FEM128 16 384 2.1416E-012 1.64 5.29 5.44

FEM256 65 536 6.7400E-012 9.73 5.93 5.22

FEM512 262 144 1.5002E-010 545.00 56.01 6.51

BEM example, smallest eigenvalues

dense matrix, but approximable by an H-matrix ⇒ MATLAB eigsis expensive, precond.: H-inversion

Name ni error titi

ti−1

N(ni )N(ni−1)

BEM8 258 7.5498E-08 0.02

BEM16 1 026 2.0455E-07 0.16 8.00 24.18

BEM32 4 098 5.4044E-07 1.33 8.31 24.22

BEM64 16 386 1.8159E-06 11.90 8.95 23.99

BEM128 65 538 — 146.00 12.27 14.66

dense matrix, but approximable by an H-matrix ⇒ MATLAB eigsis expensive, precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

BEM8 258 8.2758E-08 0.02

BEM16 1 026 1.9546E-07 0.18 9.00 24.18

BEM32 4 098 5.2752E-07 0.95 5.28 24.22

BEM64 16 386 2.1935E-06 7.66 8.06 23.99

BEM128 65 538 — 33.20 4.33 14.66

BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

H-PINVIT(1,6) O(N(ni ))H-inversion O(N(ni ) log ni )

BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

H-PINVIT(1,6) O(N(ni ))H-inversion O(N(ni ) log ni )

MATLAB eigs

BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

H-PINVIT(1,6) O(N(ni ))H-Chol. decomp. O(N(ni ) log ni )

BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

H-PINVIT(1,6) O(N(ni ))H-Chol. decomp. O(N(ni ) log ni )

MATLAB eigs

Conclusions

Once we have inverted M ∈ H(T , k) finding the eigenvaluesby H-PINVIT is cheap and storage efficient.

For dense, data-sparse matrices, H-PINVIT can solveproblems where the MATLAB function eigs (implicitly,restarted Arnoldi/Lanczos iteration) is not applicable.

The folded spectrum method enables us to compute innereigenvalues, too.

Further details in our preprint MPIMD/11-01:http://www.mpi-magdeburg.mpg.de/preprints/.

Thank you for your attention.

Conclusions

Appendix

Adaptive H-Inversion [Grasedyck 2001]

Adaptive H-Inversion

Input: M ∈ H (TI×I ), c = c√‖M‖2

Output: M−1H , with∥∥I −M−1H M

∥∥2< c

Compute M−1H with εlocal = c/(Csp(M) ‖M‖H2 )

δ0M :=∥∥I −M−1H M

∥∥H2/c

while δiM > 1 doCompute M−1H with εlocal = εlocal/δ

δiM :=∥∥I −M−1H M

∥∥H2/c

end∥∥I −M−1H M∥∥2< c = c√

‖M‖2⇔

∥∥I −M−1H M∥∥M≤ c

Appendix

Adaptive H-Cholesky Decomposition

Input: M ∈ H (TI×I ), c = c√‖M‖2

Output: LHLTH, with

∥∥∥I − L−TH L−1H M∥∥∥2< c

Compute LHLTH = M with εlocal = c/(Csp(M) ‖M‖H2 )

δ0M :=∥∥∥I − L−TH L−1H M

∥∥∥H2/c

while δiM > 1 doCompute LHL

TH = M with εlocal = εlocal/δ

δiM :=∥∥∥I − L−TH L−1H M

∥∥∥H2/c

end∥∥∥I − L−TH L−1H M∥∥∥2< c = c√

‖M‖2⇔

∥∥∥I − L−TH L−1H M∥∥∥M≤ c

Appendix

Why is it called Preconditioned Inverse Iteration?

Inverse Iteration:

xi+1 = µ(xi )A−1xi ,

xi+1 = xi − A−1Axi︸︷︷︸=0

+µ(xi )A−1xi ,

xi+1 = xi − A−1 (Axi + µ(xi )xi ) .

Replace A−1 with the inexact solution by the preconditioner B:

xi+1 = xi − B−1 (Axi + µ(xi )xi ) .

Preconditioned Inverse Iteration for Hierarchical Matrices

Education

Transcript of Preconditioned Inverse Iteration for Hierarchical Matrices

PRIMME: PReconditioned Iterative MultiMethod Eigensolver ...

FAST RADIAL BASIS FUNCTION INTERPOLATION VIA PRECONDITIONED KRYLOV ITERATION

Preconditioned low-rank Riemannian optimization for linear ...€¦ · Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure Daniel Kressner

Preconditioned Methods for Sampling Multivariate Gaussian Distributionsechow/pubs/sampling.pdf · Preconditioned Methods for Sampling Multivariate Gaussian Distributions Edmond Chow

Preconditioned Conjugate Gradient Methods in Truncated ...cjlin/papers/tron_pcg/precondition.pdf · Preconditioned Conjugate Gradient Methods in Truncated ... The model parameters,

PRECONDITIONED EXPLICIT DECOUPLED GROUP METHODS …eprints.usm.my/43006/1/Pages_from_PRECONDITIONED_EXPLICIT_DECOUPLED.pdfi PRECONDITIONED EXPLICIT DECOUPLED GROUP METHODS FOR SOLVING

Preconditioned Multigrid Methods for Compressible Flow

Preconditioned residual methods for solving steady fluid flows

Comparison of some Preconditioned Krylov Methods for

Preconditioned Inverse Iteration for Hierarchical Matrices

H2O2-preconditioned human adipose-derived stem cells ...

PRECONDITIONED CONJUGATE GRADIENT METHOD … · PRECONDITIONED CONJUGATE GRADIENT METHOD FOR OPTIMAL CONTROL PROBLEMS WITH CONTROL AND STATE CONSTRAINTS ROLAND HERZOG AND EKKEHARD

Original Article Mesenchymal stem cells preconditioned ... stem cells preconditioned with trimetazidine promote neovascularization ... Isolation and culture of ... Hanks buffer. The

Implementation of the De ated Preconditioned Conjugate ...ta.twi.tudelft.nl/users/vuik/numanal/gupta_afst.pdf · MSc THESIS COMPUTER ENGINEERING \Implementation of the De ated Preconditioned

Preconditioned Air Unit - ZEPHIR - ADELTE Air Unit - ZEPHIR Preconditioned Air Unit Point-of-Use / Stand or Bridge Mounted The flexible way to deliver preconditioned air, offering

arXiv:2002.07555v1 [math.NA] 18 Feb 2020 · 2 Gitte Kremling, Robert Speck using a preconditioned xed-point iteration. The iterative structure of SDC has been proven to provide many

CONSTRAINT-PRECONDITIONED KRYLOV SOLVERS FOR … · CahierduGERADG-2019-72 CONSTRAINT-PRECONDITIONED KRYLOV SOLVERS FOR REGULARIZED SADDLE-POINT SYSTEMS DANIELA DI SERAFINO AND DOMINIQUE

Preconditioned Iterative Methods for Linear Systems

GPU-Accelerated Preconditioned Iterative Linear Solvers

Large-Scale Topology Optimization using Preconditioned Krylov