Preconditioned Inverse Iteration for Hierarchical Matrices

MAX PLANCK INSTITUTE

FOR DYNAMICS OF COMPLEX

TECHNICAL SYSTEMS

MAGDEBURG

82nd GAMM Annual Scientific ConferenceGraz, April 19, 2011

Preconditioned Inverse Iteration forHierarchical Matrices

Peter Benner and Thomas Mach

Max Planck Institute for Dynamics of Complex Technical SystemsComputational Methods in Systems and Control Theory

Magdeburg

Max Planck Institute Magdeburg Thomas Mach, Preconditioned Inverse Iteration for Hierarchical Matrices 1/28

Hierarchical (H-)Matrices PINVIT Numerical Results

Outline

1 Hierarchical (H-)Matrices

2 PINVIT

3 Numerical Results



H-Matrices [Hackbusch 1998]

Some dense matrices, e.g. BEM or FEM, can be approximated byH-matrices in a data-sparse manner.

hierarchical tree TI block H-tree TI× I

I = {1, 2, 3, 4, 5, 6, 7, 8}

{1, 2, 3, 4} {5, 6, 7, 8}

{1, 2} {3, 4} {5, 6} {7, 8}

{1}{2}{3}{4}{5}{6}{7}{8}

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 56 6 6 67 7 7 78 8 8 8

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 56 6 6 67 7 7 78 8 8 8

dense matrices, rank-k-matrices

rank-k-matrix: Ms×t = ABT , A ∈ Rn×k ,B ∈ Rm×k (k � n,m)




Hierarchical matrices

H(TI× I, k) ={M ∈ RI× I

∣∣ rank (Ma×b) ≤ k ∀a× b admissible}

22

3 3

7 10

3 7

3 10

19 10

10 31

14 8

11 11

14 118 11

19 10

10 31 11

11 31

11 9

9 16 12

11 1611 8

9 16 11

11 16

619 73 3

8

11 11

9 3

7 3

11

8 11

25 10

10 19 11

11 31

8 5

11 8

8 15 8 12

116 515 6

13

13

7 5

13 8

8 11 8 11

116 5

15 6

8

11 8

8 15

5 8 11

126 15

5 6 13

137

13 8

8 11

5 8 11

116 15

5 6

6110 103

6 14

10 3 6

10 14

25 10 6

10

6

19 10

10 31

15 9

9 11 10

10 1615 9

8 11 10

10 16

5110 107 9 7

3 3

10 7

109 3

7 3

25 11

1125 10

10 19

11 811 8

8 15 9

8 15 12 13

1310 7

13 8

8 11 9

9 15 1119

2010 7

13 8

9

11 8

8 15 11 12

139 7

13 8

9

13 8

8 1111

1011 8

8 15 8

9 15

7 12 13

1310

13 8

8 11 8

9 15

7 1120

199

13 9

8

11 8

8 15

7 11 12

129

13 9

813 8

8 11

7 11

39 10

10 25

3 7

3 10 107 10 6

3 3

7 10 7

1010

6

22

3 3

7 10

3 7

3 10

19 10

10 31

15 10

11

11 9

9 16

15 11

1011 8

9 16

34 10

10 25

13 10

7 1113 7

10 11 61

6 513 6 11

12

8 5

11 8

8 15 812

136 5

13 6 11

11 236 13

5 6 12

118

11 8

8 15

5 813

12

6 13

5 6 10

11 23

20 9

9 39

9 7

103 7

3 10

9 107

3 3

7 10 61

15 10

1015 9

9 11

15 10

1015 9

8 11

20 9

9 34

9 7

10 13

9 107 13 51

adaptive rank k(ε)

storage NSt,H(T , k) = O(n log n k(ε))

complexity of approximate arithmetic

MHv O(n log n k(ε))+H,−H O(n log n k(ε)2)

∗H,HLU(·), (·)−1H O(n (log n)2 k(ε)2)




Hierarchical matrices

H(TI× I, k) ={M ∈ RI× I

∣∣ rank (Ma×b) ≤ k ∀a× b admissible}

22

3 3

7 10

3 7

3 10

19 10

10 31

14 8

11 11

14 118 11

19 10

10 31 11

11 31

11 9

9 16 12

11 1611 8

9 16 11

11 16

619 73 3

8

11 11

9 3

7 3

11

8 11

25 10

10 19 11

11 31

8 5

11 8

8 15 8 12

116 515 6

13

13

7 5

13 8

8 11 8 11

116 5

15 6

8

11 8

8 15

5 8 11

126 15

5 6 13

137

13 8

8 11

5 8 11

116 15

5 6

6110 103

6 14

10 3 6

10 14

25 10 6

10

6

19 10

10 31

15 9

9 11 10

10 1615 9

8 11 10

10 16

5110 107 9 7

3 3

10 7

109 3

7 3

25 11

1125 10

10 19

11 811 8

8 15 9

8 15 12 13

1310 7

13 8

8 11 9

9 15 1119

2010 7

13 8

9

11 8

8 15 11 12

139 7

13 8

9

13 8

8 1111

1011 8

8 15 8

9 15

7 12 13

1310

13 8

8 11 8

9 15

7 1120

199

13 9

8

11 8

8 15

7 11 12

129

13 9

813 8

8 11

7 11

39 10

10 25

3 7

3 10 107 10 6

3 3

7 10 7

1010

6

22

3 3

7 10

3 7

3 10

19 10

10 31

15 10

11

11 9

9 16

15 11

1011 8

9 16

34 10

10 25

13 10

7 1113 7

10 11 61

6 513 6 11

12

8 5

11 8

8 15 812

136 5

13 6 11

11 236 13

5 6 12

118

11 8

8 15

5 813

12

6 13

5 6 10

11 23

20 9

9 39

9 7

103 7

3 10

9 107

3 3

7 10 61

15 10

1015 9

9 11

15 10

1015 9

8 11

20 9

9 34

9 7

10 13

9 107 13 51

adaptive rank k(ε)

storage NSt,H(T , k) = O(n log n k(ε))

complexity of approximate arithmetic

MHv O(n log n k(ε))+H,−H O(n log n k(ε)2)

∗H,HLU(·), (·)−1H O(n (log n)2 k(ε)2)

A1

BT1

+ A2

BT2

= A1 A2

BT1

BT2



Goal: Eigenvalues of symmetric H-Matrices

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λi in O(n (log n)α kβ)?

X

How to find λi in O(n (log n)α kβ)?

X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

X


X



Preconditioned Inverse Iteration[Knyazev, Neymeyr, et al.]

Definition

The function

µ(x) = µ(x ,M) =xTMx

xT x

is called the Rayleigh quotient.

Minimize the Rayleigh quotient by a gradient method:

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xT x(Mx − xµ(x)) ,

+ preconditioning ⇒ update equation:

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Residual r(x) = Mx − xµ(x).

Preconditioned residual B−1r(x) = B−1 (Mx − xµ(x)).



Preconditioned Inverse Iteration[Knyazev, Neymeyr 2009]

xi+1 := xi − B−1 (Mxi − xiµ(xi ))

If

M ∈ Rn×n symmetric positive definite and

B−1 approximates the inverse of M, so that∥∥I − B−1M∥∥M≤ c < 1,

then Preconditioned INVerse ITeration (PINVIT) converges andthe number of iterations is independent of n.



Preconditioned Inverse Iteration[Knyazev, Neymeyr, et al.]

The residual

ri = Mxi − xiµ(xi )

converges to 0, so that

‖ri‖2 < ε

is a useful termination criterion.



Variants of PINVIT[Neymeyr 2001: A Hierarchy of Precond. Eigens. for Ellipt. Diff. Op.]

Classification by Neymeyr:

PINVIT(1): xi+1 := xi − B−1ri .

PINVIT(2): xi+1 := arg minv∈span{xi ,B−1ri} µ(v).

PINVIT(3): xi+1 := arg minv∈span{xi−1,xi ,B−1ri} µ(v).

PINVIT(n): Analogously.

PINVIT(·,d): Replacing x by a rectangular full rank matrixX ∈ Rn×d one gets the subspace version of PINVIT(·).



Algorithm and Complexity

The number of iterations is independent of matrix size n.

H-PINVIT(1,d)

Input: M ∈ Rn×n, X0 ∈ Rn×d (XT0 X0 = I , e.g. randomly chosen)

Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1H or B−1 = L−TH L−1H

O(n (log n)2 k (c)2)

R := MX0 − X0µ, µ = XT0 MX0

for (i := 1;‖R‖F > ε; i + +) do

Xi := Orthogonalize(Xi−1 − B−1R

)

O(n (log n) k (c)2)

R := MXi − Xiµ, µ = XTi MXi

O(n (log n) k (c))

end

The complexity of the algorithm is determined by the H-matrixinversion/Cholesky decomposition: ⇒ O(n (log n)2 k (c)2).

Competitive to MATLAB® eigs.

Expensive.





H-PINVIT(1,d)


Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1H or B−1 = L−TH L−1H O(n (log n)2 k (c)2)R := MX0 − X0µ, µ = XT

0 MX0

for (i := 1;‖R‖F > ε; i + +) do


)O(n (log n) k (c)2)

R := MXi − Xiµ, µ = XTi MXi O(n (log n) k (c))

end


Competitive to MATLAB eigs.

Expensive.




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

X


X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)? X


X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0



X



Folded Spectrum Method


If i = n − d with d < O(log n),

use subspace version PINVIT(·,d).

0λn λn−1λn−2. . .

λ1

But (M − σI) is not positive definite.





If i = n − d with d � log n?

shift with σ near λi .

0λn. . .

λi+1 λi λi−1. . .

λ1






If i = n − d with d � log n,

shift with σ near λi .

0λn. . .

λi+1 λi λi−1. . .

λ1σ





Folded Spectrum Method [Wang, Zunger 1994]

Mσ = (M − σI)2

Mσ is s.p.d., if M is s.p.d. and σ 6= λi .

Assume all eigenvalues of Mσ are simple.

Mv = λv ⇔ Mσv = (M − σI)2 v

= M2v − 2σMv + σ2v

= λ2v − 2σλv + σ2v

= (λ− σ)2v




1 Choose σ.2 Compute

a) Mσ = (M − σI)2 andb) M−1

σ or LLT = Mσ.

3 Use PINVIT to find the smallest eigenpair (µσ, v) of Mσ.4 Compute µ = vTMv/vTv .

(µ, v) is the nearest eigenpair to σ.

H-arithmetic enables us to shift, square and invert M resp.Mσ in linear-polylogarithmic complexity.

If M is sparse, then shifting, squaring and inverting isprohibitive.




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0



X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0


How to find λi in O(n (log n)α kβ)? X



Numerical Results

Numerical Results



Hlib

Hlib [Borm, Grasedyck, et al.]

We use the Hlib1.3 (www.hlib.org) for theH-arithmetic operations and some examples out ofthe library for testing the eigenvalue algorithm.


www.hlib.org


2D Laplace, smallest eigenvalues

2D Laplace over [−1, 1]× [−1, 1], precond.: H-inversion

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 5.6146E-010 0.01

FEM16 256 4.5918E-010 0.02 2.00 106.67

FEM32 1 024 3.7550E-010 0.12 6.17 27.08

FEM64 4 096 3.8009E-010 0.82 6.68 8.06

FEM128 16 384 4.4099E-010 5.84 7.09 5.44

FEM256 65 536 3.9651E-010 34.47 5.91 5.22

FEM512 262 144 3.7877E-010 194.00 5.63 6.51

d = 4, c = 0.2, εeig = 10−4, N(ni ) = ni (log2 ni )CspCid

Time t only H-PINVIT (without H-inversion)

∥∥∥∥∥∥∥∥∥

λ1 − λ1λ2 − λ2λ3 − λ3λ4 − λ4

∥∥∥∥∥∥∥∥∥2




FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

100

101

102

103

104

CP

Uti

me

ins

H-PINVIT(1,4) H-inversion

H-PINVIT(3,4) O(N(ni ) log ni )

O(N(ni ))




FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

100

101

102

103

104

CP

Uti

me

ins

H-PINVIT(1,4) H-inversion


O(N(ni )) MATLAB eigs





H-PINVIT(1,d)


Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1H or B−1 = L−TH L−1H O(n (log n)2 k (c)2)R := MX0 − X0µ, µ = XT

0 MX0

for (i := 1;‖R‖F > ε; i + +) do


)O(n (log n) k (c)2)

R := MXi − Xiµ, µ = XTi MXi O(n (log n) k (c))

end


Competitive to MATLAB eigs.

Expensive.




2D Laplace over [−1, 1]× [−1, 1], precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 4.6920E-010 0.01

FEM16 256 4.7963E-010 0.02 2.00 106.67

FEM32 1 024 3.4696E-010 0.08 4.00 27.08

FEM64 4 096 4.6414E-010 0.48 6.00 8.06

FEM128 16 384 3.3206E-010 3.20 6.67 5.44

FEM256 65 536 3.8468E-010 13.90 4.34 5.22

FEM512 262 144 3.1353E-010 62.40 4.49 6.51


Time t only H-PINVIT (without H-Cholesky decomposition)

∥∥∥∥∥∥∥∥∥

λ1 − λ1λ2 − λ2λ3 − λ3λ4 − λ4

∥∥∥∥∥∥∥∥∥2




FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

100

101

102

103

104

CP

Uti

me

ins

H-PINVIT(1,4) H-Chol. decomp.


O(N(ni ))




FEM16FEM32

FEM64FEM128

FEM256FEM512

10−3

10−2

10−1

100

101

102

103

104

CP

Uti

me

ins

H-PINVIT(1,4) H-Chol. decomp.


O(N(ni )) MATLAB eigs



2D Laplace, vn, FEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



2D Laplace, vn−1, FEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2




−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



2D Laplace, inner eigenvalues

2D Laplace over [−1, 1]× [−1, 1], precond.: H-inversion

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 1.7219E-013 <0.01

FEM16 256 7.2388E-012 0.03 106.67

FEM32 1 024 6.0045E-013 0.12 3.57 27.08

FEM64 4 096 1.0915E-012 0.93 7.48 8.06

FEM128 16 384 1.5655E-012 6.08 6.50 5.44

FEM256 65 536 1.1976E-011 52.40 8.62 5.22





2D Laplace, inner eigenvalues

2D Laplace over [−1, 1]× [−1, 1], precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 1.5306E-013 <0.01

FEM16 256 9.8071E-012 0.02 106.67

FEM32 1 024 1.0577E-012 0.05 2.50 27.08

FEM64 4 096 1.5776E-012 0.31 6.20 8.06

FEM128 16 384 2.1416E-012 1.64 5.29 5.44

FEM256 65 536 6.7400E-012 9.73 5.93 5.22

FEM512 262 144 1.5002E-010 545.00 56.01 6.51





BEM example, smallest eigenvalues

dense matrix, but approximable by an H-matrix ⇒ MATLAB eigsis expensive, precond.: H-inversion

Name ni error titi

ti−1

N(ni )N(ni−1)

BEM8 258 7.5498E-08 0.02

BEM16 1 026 2.0455E-07 0.16 8.00 24.18

BEM32 4 098 5.4044E-07 1.33 8.31 24.22

BEM64 16 386 1.8159E-06 11.90 8.95 23.99

BEM128 65 538 — 146.00 12.27 14.66






dense matrix, but approximable by an H-matrix ⇒ MATLAB eigsis expensive, precond.: H-Cholesky decomp.

Name ni error titi

ti−1

N(ni )N(ni−1)

BEM8 258 8.2758E-08 0.02

BEM16 1 026 1.9546E-07 0.18 9.00 24.18

BEM32 4 098 5.2752E-07 0.95 5.28 24.22

BEM64 16 386 2.1935E-06 7.66 8.06 23.99

BEM128 65 538 — 33.20 4.33 14.66






BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

100

101

102

103

104

105

CP

Uti

me

ins

H-PINVIT(1,6) O(N(ni ))H-inversion O(N(ni ) log ni )




BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

100

101

102

103

104

105

CP

Uti

me

ins

H-PINVIT(1,6) O(N(ni ))H-inversion O(N(ni ) log ni )

MATLAB eigs




BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

100

101

102

103

104

105

CP

Uti

me

ins

H-PINVIT(1,6) O(N(ni ))H-Chol. decomp. O(N(ni ) log ni )




BEM8BEM16

BEM32BEM64

BEM12810−3

10−2

10−1

100

101

102

103

104

105

CP

Uti

me

ins

H-PINVIT(1,6) O(N(ni ))H-Chol. decomp. O(N(ni ) log ni )

MATLAB eigs



Conclusions

Once we have inverted M ∈ H(T , k) finding the eigenvaluesby H-PINVIT is cheap and storage efficient.

For dense, data-sparse matrices, H-PINVIT can solveproblems where the MATLAB function eigs (implicitly,restarted Arnoldi/Lanczos iteration) is not applicable.

The folded spectrum method enables us to compute innereigenvalues, too.

Further details in our preprint MPIMD/11-01:http://www.mpi-magdeburg.mpg.de/preprints/.

Thank you for your attention.


h

Appendix

Adaptive H-Inversion [Grasedyck 2001]

Adaptive H-Inversion

Input: M ∈ H (TI×I ), c = c√‖M‖2

∈ R

Output: M−1H , with∥∥I −M−1H M

∥∥2< c

Compute M−1H with εlocal = c/(Csp(M) ‖M‖H2 )

δ0M :=∥∥I −M−1H M

∥∥H2/c

while δiM > 1 doCompute M−1H with εlocal = εlocal/δ

iM

δiM :=∥∥I −M−1H M

∥∥H2/c

end∥∥I −M−1H M∥∥2< c = c√

‖M‖2⇔

∥∥I −M−1H M∥∥M≤ c


Appendix

Adaptive H-Cholesky Decomposition

Adaptive H-Cholesky Decomposition

Input: M ∈ H (TI×I ), c = c√‖M‖2

∈ R

Output: LHLTH, with

∥∥∥I − L−TH L−1H M∥∥∥2< c

Compute LHLTH = M with εlocal = c/(Csp(M) ‖M‖H2 )

δ0M :=∥∥∥I − L−TH L−1H M

∥∥∥H2/c

while δiM > 1 doCompute LHL

TH = M with εlocal = εlocal/δ

iM

δiM :=∥∥∥I − L−TH L−1H M

∥∥∥H2/c

end∥∥∥I − L−TH L−1H M∥∥∥2< c = c√

‖M‖2⇔

∥∥∥I − L−TH L−1H M∥∥∥M≤ c


Appendix

Why is it called Preconditioned Inverse Iteration?

Inverse Iteration:

xi+1 = µ(xi )A−1xi ,

xi+1 = xi − A−1Axi︸︷︷︸=0

+µ(xi )A−1xi ,

xi+1 = xi − A−1 (Axi + µ(xi )xi ) .

Replace A−1 with the inexact solution by the preconditioner B:

xi+1 = xi − B−1 (Axi + µ(xi )xi ) .


Preconditioned Inverse Iteration for Hierarchical Matrices

Education

Transcript of Preconditioned Inverse Iteration for Hierarchical Matrices