Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute,...

51
Adaptive Multilevel BDDC Jan Mandel Includes joint work with Clark Dohrmann, Bedˇ rich Soused´ ık, Jakub ˇ ıstek, Pavel Burda, Marta ˇ Cert´ ıkov´ a, and Jaroslav Novotn´ y. Center for Computational Mathematics and Department of Mathematical and Statistical Sciences University of Colorado Denver Supported by National Science Foundation under grant DMS-0713876 Max-Planck Institute, Leipzig, March 19, 2010 (Leipzig) Adaptive Multilevel BDDC Max-Planck Institute, Leipzig, March 19, 20 / 49

Transcript of Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute,...

Page 1: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Adaptive Multilevel BDDC

Jan Mandel

Includes joint work with Clark Dohrmann, Bedrich Sousedık, JakubSıstek, Pavel Burda, Marta Certıkova, and Jaroslav Novotny.

Center for Computational Mathematicsand

Department of Mathematical and Statistical SciencesUniversity of Colorado Denver

Supported by National Science Foundation under grant DMS-0713876

Max-Planck Institute, Leipzig, March 19, 2010

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 1

/ 49

Page 2: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Outline

Some (biased) history

BDDC formulation

Adaptive Multilevel BDDC

Implementation by global matrices and generalized change of variables

Implementation on top of frontal solver

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 2

/ 49

Page 3: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Motivation

BDDC – Balancing Domain Decomposition by Constraints

From Dohrmann (2003):

Some remaining issues need to be addressed for improvement.First, it would be useful to have an effective method for select-ing additional corners and edges to improve performance for verypoorly conditioned problems. Second, the performance of themultilevel extension should be investigated further. Recall thatthe multilevel extension is obtained by recursive application of thepreconditioner to coarse problem stiffness matrices. Such an ex-tension would be beneficial for problems with very large numbersof substructures...

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 3

/ 49

Page 4: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Basic substructuring

assemble elements into substructures, eliminate interiors =⇒ reducedproblem (Schur complement) on interface (Dirichlet-to-Neumannoperator H1/2 → H−1/2, a.k.a. Poincare-Steklov operator)

for parallel solution; Schur complement matrix-vector multiply =solve substructure Dirichlet problem

only matrix datastructures needed; condition number O(1/h) betterin practice than the O

�1/h2

�for the original problem (for small N)

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 4

/ 49

Page 5: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Early Preconditioners

diagonal preconditioning: Gropp and Keyes 1987, “probing” thediagonal of the Schur complement Chan and Mathew 1992 (becausecreating the diagonal of the Schur complement is expensive)...

preconditioning by solving substructure Neumann problems(H−1/2 → H1/2) Glowinski and Wheeler 1988, Le Talleck and DeRoeck 1991 (a.k.a. the Neumann-Neumann method)

add coarse space – asymptotically optimal preconditioners:H/h → log2 (1 + H/h) (Bramble, Pasciak, Schatz 1986+, Widlund1987, Dryja 1988, Dryja-Widlund-Smith 1994... ) but all thesemethods require access to mesh details and depend on detailsof the Finite Element code, which makes them hard tointerface with existing FE software

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 5

/ 49

Page 6: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

FETI and BDD

algebraic - need only substructure 1. solvers (Neumann, Dirichlet), 2.connectivity, 3. basis of nullspace (BDD - constant function for theLaplacian, FETI - actual nullspace)

both involve singular substructure problems (Neumann) and build thecoarse problem from local substructure nullspaces (in different ways)to assure that the singular systems are consistent

Balancing Domain Decomposition (BDD, Mandel 1993): solve thesystem reduced to interfaces, interface degrees of freedom common(this is the Neumann-Neumann with a particular coarse space andmultiplicative coarse correction)

Finite Element Tearing and Interconnecting (FETI, Farhat and Roux1991): enforce continuity across interfaces by Lagrange multipliers,solve the dual system for the multipliers

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 6

/ 49

Page 7: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

FETI and BDD Developments

Both require only matrix level information available in FE software.So they became very popular and widely used. The methods workwell in 2D and 3D (solids).

But the performance for plates/shells/biharmonic was not so good.Reason: the condition numbers depend on the energy (trace norm) offunctions with jumps across a substructure corner. In 2D, OK forH1/2 traces of H1 functions, not H3/2 traces of H2 functions(embedding theorem).

Fix: avoid this by increasing the coarse space and so restricting thespace where the method runs, to make sure that nothing gets tornacross the corners (BDD: LeTallec Mandel Vidrascu 1998, FETI:Farhat Mandel 1998, Farhat Mandel Tezaur 1998). Drawback:complicated, expensive, a large coarse problem with custom basisfunctions

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 7

/ 49

Page 8: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

The best so far:

FETI-DP and BDDC

To assure that nothing gets torn across the corners, enforce identicalvalues at corners from all neighboring substructures a-priori =⇒corner values are coarse degrees of freedom

Continuity elsewhere at the interfaces by Lagrange multipliers as inFETI =⇒ FETI-DP (Farhat et al 2001)

Continuity elsewhere by common values as in BDD =⇒ BDDC(Dohrmann 2003; independently Cros 2003, Frakagis andPapadrakakis 2003, with corner coarse degrees of freedom only)

Additional coarse degrees of freedom (side/face averages)required in 3D for good conditioning: Farhat, Lesoinne, Pierson 2000(algorithm only), Dryja Widlud 2002 (with proofs)

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 8

/ 49

Page 9: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Evolution of BDD/BDDC/FETI-DP

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 9

/ 49

Page 10: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Substructuring for the two-level method (with H/h=4)

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 10

/ 49

Page 11: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

BDDC description - example of spaces

W =N�i=1

Wi : space of block vectors, one block per substructure

U ⊂ �W ⊂ Wcontinuous across whole continuous across no continuitysubstructure interfaces corners only requiredglobal matrix: assembled partially assembled not assembledsubstructures: assembled assembled assembled

Want to solveu ∈ U : a (u, v) = �f , v� ∀v ∈ U.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 11

/ 49

Page 12: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Abstract two-level BDDC:Variational setting of the problem and algorithm components

u ∈ U : a(u, v) = �f , v� , ∀v ∈ U

form a (·, ·) is SPD on U and positive semidefinite on W ⊃ U,Example:W = W1 × · · ·×WN (spaces on substructures)U = functions continuous across interfaces

Choose preconditioner components:1 space �W , U ⊂ �W ⊂ W , such that a (·, ·) is positive definite on �W .

Example: functions with continuous coarse dofs, such as values atsubstructure corners

2 projection E : �W → U, rangeE = U .Example: averaging across substructure interfaces

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 12

/ 49

Page 13: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Abstract BDDC preconditioner

Theorem

The abstract BDDC preconditioner M : U −→ U defined by

M : r �−→ u = Ew , w ∈ �W : a (w , z) = �r ,Ez� , ∀z ∈ �W .

satisfies

κ =λmax(MA)

λmin(MA)≤ ω = sup

w∈�W

�(I − E )w�2a

�w�2a

.

The space �W is defined using so called coarse degrees of freedom, as

�W = {w ∈ W : C (I − E )w = 0} ,

C . . . weights on the local coarse degrees of freedom, E . . . averaging,

=⇒coarse degrees of freedom on adjacent substructures coincide.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 13

/ 49

Page 14: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Coarse degrees of freedom (in implementation)

We want to represent common values of coarse dofs as global coarse dofs,QP . . . the global coarse degrees of freedom selection matrix.Suppose there is a space Uc and operators

QTP : U → Uc Rc : Uc → X R : U → W ,

where Rc and R are mapping (0− 1) matrices, related as CR = RcQTP .

In implementation, �W is decomposed into

�W = �W∆ ⊕ �WΠ

�W∆ = functions with zero coarse dofs ⇒ local problems on substructures�WΠ = functions given by coarse dofs & energy minimal ⇒

⇒ global coarse problem.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 14

/ 49

Page 15: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

The coarse problem

0

0.2

0.4

0.6

0.8

1

A basis function from �WΠ is energy

minimal subject to given values of

coarse degrees of freedom on the

substructure. The function is discon-

tinuous across the interfaces between

the substructures but the values of

coarse degrees of freedom on the dif-

ferent substructures coincide.

The coarse problem has the same structure as the original FE problem =⇒solve it approximately by one iteration of BDDC =⇒ three-level BDDC.

Apply recursively =⇒ Multilevel BDDC.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 15

/ 49

Page 16: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Substructuring for the three-level method

hH1

H2

1,iΩ

2,kΩ

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 16

/ 49

Page 17: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Multilevel BDDC

Coarse problem solved by the BDDC preconditioner recursively.U�

UI1P1←⊂ U1

E1←⊂

�W1

��WΠ1 ⊕ �W∆1

�UI2

P2←⊂ U2

E2←⊂

�W2

��WΠ2 ⊕ �W∆2

�...�

UIL−1PL−1←⊂ UL−1

EL−1←⊂

�WL−1

��WΠL−1 ⊕ �W∆L−1

The leaves of the tree are the local problems and the coarse actually solved.(Leipzig) Adaptive Multilevel BDDC

Max-Planck Institute, Leipzig, March 19, 2010 17/ 49

Page 18: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Condition number bound

Theorem (Mandel, Sousedık, Dohrmann (2008))

The condition number bound κ ≤ ω of Multilevel BDDC is given by

ω = ΠL−1�=1ω� , ω� = sup

w�∈(I−P�)�W�

�(I − E�)w��2a

�w��2a

Generalizes 3-level bounds by Tu (2006, 2007) to many levels.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 18

/ 49

Page 19: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Adaptive method idea (Mandel, Sousedik, 2006)

Condition number bound is Rayeigh quotient

ω� ≤ supw∈(I−P�)�W�

�(I − E�)w��2a

�w��2a

= supw�∈�W�

wT� (I − E�)

T S� (I − E�)w�

wT� S�w�

Eigenvalues λ1 > λ2 > . . ., stationary points are eigenvectors. Optimaldecrease of the Rayleigh quotient: make the space

�W� = kerC (I − E�)

smaller by orthogonality to the dominant eigenvector u1:add uT1 to C (I − E�)This reduces the condition bound to the second eigenvalue λ2.Adding more eigenvectors eats more eigenvalues.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 19

/ 49

Page 20: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

The condition number bound as an eigenvalueproblem

On the level �, define λ� by

Π� (I − E�)T S� (I − E�)Π�w� = λ�Π�S�Π�w�,

whereS� . . . the Schur complement operator,Π� . . . the orthogonal projection in (I − P�)W� into (I − P�)�W�.

Thenκ ≤ ω = ΠL−1

�=1 maxλ�.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 20

/ 49

Page 21: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Condition number bound: Heuristic indicator

Solving the the global eigenvalue problems is expensive ... replace it by:local versions formulated for all pairs of adjacent substructures.

Definition: A pair of substructures is adjacent, if they share a face.

ωst� = maxλst

� : Πst�

�I − E st

�TS st�

�I − E st

�Πst� w

st� = λst

� Πst� S

st� Πst

� wst� .

The heuristic indicator of the condition number bound is defined as

�ω = ΠL−1�=1

�max

{st}∈A�

ωst�

�.

The vectors w st� are used to generate constraints in C� and QP,� such that

�W st� =

�w st� ∈ W st

� : C st�

�I − E st

�w st� = 0

�.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 21

/ 49

Page 22: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Adaptive selection of face constraints

Algorithm: adding of coarse degrees of freedom to guarantee that thecondition number indicator �ω ≤ τL−1, for a given a target value τ :

for levels � = 1 : L− 1

for all faces F� on level �

1 Compute the largest local eigenvalues and corresponding eigenvectors,until the first mst is found such that λst

mst ≤ τ , put k = 1, . . . ,mst .

2 Compute the constraint weights cstk = [csk ctk ] as

cstk = w stTk

�I − E st

�TS st�

�I − E st

�,

3 Take one block, e.g., csk and keep nonzero weights for the face F�.

4 Add to global coarse dofs selection matrix QP,� the k columns

RsT� csTk .

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 22

/ 49

Page 23: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Adaptive-Multilevel BDDC: Implementation remarks

Reduce problem to interfaces on level � = 1,on levels � > 1 apply interior pre/post-corrections in iterations.

Use the adaptive method for faces, with initial constraints as:

corners in 2D,corners and arithmetic averages over edges in 3D.

Treat substructures as (coarse) elements with:energy minimal basis functions,variable number of nodes per element,variable number of degrees of freedom per node.

On each decomposition level � = 1, . . . , L− 1:create substructures with roughly the same number of dofs,minimize the number of “cuts“ (Metis 4) between substructures,repeat until it is suitable to factor the coarse problem directly (level L).

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 23

/ 49

Page 24: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 2D: compressible elasticity, λ = 1, µ = 2

Domain decomposition of the planar elasticity problem with 1182722 dof,2304 subdomains on the second level and 9 subdomains on the third-level,

The two-level decomposition (left) and the three-level decomposition (right):

The coarsening ratio on both decomposition levels is Hi/Hi−1 = 16.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 24

/ 49

Page 25: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 2D: Adaptive 2-level method

Non-adaptive method:constraint Nc C κ it

c 4794 9.3 18.41 43c+f 13818 26.9 18.43 32

Adaptive constraints:τ Nc C �ω κ it

∞(=c) 4794 9.3 - 18.41 4310 4805 9.4 8.67 8.34 343 18110 35.3 2.67 2.44 152 18305 35.7 1.97 1.97 13

c, c+f: constraints as arithmetic averages over corners, corners and faces,Nc : number of constraints, C: relative size of the coarse problem,τ : condition number target, �ω: condition number indicator,κ: approximate condition estimate, it: number of iterations (tol 10−8).

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 25

/ 49

Page 26: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 2D: Local eigenvalues (2-level method)

Eigenvalues of the local problems for pairs of subdomains i and j :

(the jagged face is between subdomains 2 and 50)

i j λij,1 λij,2 λij,3 λij,4 λij,5 λij,6 λij,7 λij,8

1 2 3.8 2.4 1.4 1.3 1.2 1.1 1.1 1.11 49 6.0 3.5 2.7 1.4 1.3 1.1 1.1 1.12 3 5.4 2.6 1.6 1.3 1.2 1.1 1.1 1.12 50 24.3 18.4 18.3 16.7 16.7 14.7 13.5 13.13 4 3.4 2.4 1.4 1.3 1.1 1.1 1.1 1.13 51 7.4 4.6 3.7 1.7 1.4 1.3 1.2 1.149 50 12.6 5.1 4.3 1.9 1.6 1.3 1.2 1.250 51 8.7 4.8 3.9 1.8 1.5 1.3 1.2 1.250 98 7.5 4.6 3.7 1.7 1.4 1.3 1.2 1.1

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 26

/ 49

Page 27: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 2D: Local eigenvalues (3-level method)

Eigenvalues of the local problems for pairs of subdomains i and j :

(the jagged face is between subdomains 2 and 5)

i j λij,1 λij,2 λij,3 λij,4 λij,5 λij,6 λij,7 λij,8

1 2 16.5 9.0 5.4 2.6 2.1 1.4 1.3 1.31 4 6.5 4.7 1.9 1.7 1.3 1.2 1.2 1.12 3 23.1 9.4 4.6 3.2 2.1 1.6 1.4 1.32 5 84.3 61.4 61.4 55.9 55.8 49.3 48.0 46.93 6 13.7 8.8 4.4 2.2 1.9 1.4 1.3 1.24 7 6.5 4.7 1.9 1.7 1.3 1.2 1.2 1.15 6 18.9 13.1 11.3 3.8 2.6 2.1 1.9 1.55 8 17.3 12.9 10.8 3.6 2.3 2.0 1.8 1.48 9 13.7 8.8 4.4 2.2 1.9 1.4 1.3 1.2

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 27

/ 49

Page 28: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 2D: Adaptive 3-level method

Non-adaptive method:constraint Nc C κ it

c 4794 + 24 1.0 67.5 74c+f 13818 + 48 3.0 97.7 70

Adaptive constraints:τ Nc C �ω κ it

∞(=c) 4794 + 24 1.0 - 67.5 7410 4805 + 34 1.0 > (9.80)2 37.42 603 18110 + 93 3.9 > (2.95)2 3.11 192 18305 + 117 4.0 > (1.97)2 2.28 15

c, c+f: constraints as arithmetic averages over corners, corners and faces,Nc : number of constraints, C: relative size of the coarse problem,τ : condition number target, �ω: condition number indicator,κ: approximate condition estimate, it: number of iterations (tol 10−8).

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 28

/ 49

Page 29: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

What next in adaptive multilevel BDDC?

In progress:

parallel implementation

further validation on practical engineering problems

Implementation issues:

The adaptive algorithm may give many additional face coarse degreesof freedom on few substructure interfaces.

How to implement efficiently BDDC with a variable andsometime large number of coarse degrees of freedom onsubstructure faces?

Implementation by a generalized change of variableImplementation on top of frontal solver

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 29

/ 49

Page 30: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Change of variables on face or edge

Li and Widlund (2007) change of variables on an face/edge into

one constant over the face/edge

all others with zero average

In the new variables, the Implementation can then treat averages just likecorners. The subdomain matrix gets a little more dense.

old =

1 −1 . . . −11 1

. . .1 1

∗ new

One old variable needs to be chosen to be replaced by the constantfunction. Here it was the first one; any other one would be just as good.But what to do when there are more averages, with general weights, notjust 1s?

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 30

/ 49

Page 31: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Implementation: Generalized change of variables

Adaptive BDDC in 3D, Mandel, Sousedık, Sıstek (in preparation)

Permute variables so that those be replaced by averages are the first k

Transform the variables by

new =

�Bavg

0 I

�∗ old =

vT1...vTk0 I

∗ old =

�U V0 I

�∗ old

Rows of Bavg are linearly independent. The inverse transformationneeds to be stable - need U well conditioned.

QR decomposition with pivoting permutes linearly independentcolumns of Bavg first:

Bavg ∗ permutation = QR = Q[triangular rectangular ] = [U V ]

Drop the averages where diagonal entries of R small.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 31

/ 49

Page 32: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Implementation: Global matrices

All BDDC/FETI-DP operators implemented as matrices on a virtualdisconnected mesh, assembled at corners only

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 32

/ 49

Page 33: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Implementation: Global matrices (2)

All BDDC/FETI-DP operators implemented as matrices on a virtualdisconnected mesh, assembled at corners only

Operations on disconnected mesh vectors by the usual matrix-vectoralgebra

in a real implementation: by substructures, take advantage of thespecial form for speedupfor testing: just use Matlab sparse matrices and write the formulas asusual

parallel implementation of M−1u by multifrontal method withordering: MUMPS

averages by transformation of variable, or projection - still OK sparsity

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 33

/ 49

Page 34: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Bridge test problem

Finite element discretization and substructuring of the bridge construction,consisting of 157 356 degrees of freedom, 16 substructures, 250 corners, 30edges and 43 faces.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 34

/ 49

Page 35: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Adaptive performance for the bridge test problem

constraint Nc κ itc 0 2 301.4 224

c+e 180 2 252.4 220c+e+f 309 653.6 160

c+e+f (3eigv) 309 177.8 103

τ �ω Nc κ it∞(=c+e) 6 500.5 180 2 252.4 220

650 589.3 185 483.5 16930 29.6 292 28.7 645 > 5 655 5.0 262 > 2 1301 2.0 14

τ = threshold for condition indicatorω = the condition number indicator achievedNc = number of coarse dofsκ = actual condition number as estimated by PCG

it = number of iterations

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 35

/ 49

Page 36: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

BDDC Implementation on Top of Frontal Solver

Sıstek, Novotny, Mandel, Certıkova, Burda (in print)Frontal solver: Solve linear equations from finite elements with somevariables free and some constrained. Matrix given as a collection of localelement matrices, never assembled whole.In: f1 x2 Out: x1 Rea

�A11 A12

A21 A22

� �x1x2

�=

�f10

�+

�0

Rea

Straightforward: use the frontal solver to:- solve the coarse problem in BDDC (substructure treated as element)- solve the local substructure problems in BDDC in the case of cornerconstraints onlyNew: by specially crafted calls and few extras, use the frontal solver to- solve the local substructure problems in BDDC in the case of generalconstraints (averages)- build the coarse basis functions

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 36

/ 49

Page 37: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Frontal solver

B. M. Irons, 1970

direct solver for sparse matrices arising in FEM

number of flops O(n · nfron2), where nfron << n is the front width

memory demand – nfron2 if out-of-core

element-by-element approach – element matrices read from file untilwhole line is assembled, then immediately eliminated

basic scheme – block 1 - ‘free’ variables, block 2 - ‘constrained’ (also‘fixed’) variables

�A11 A12

A21 A22

� �x1x2

�=

�f1f2

�+

�0

Rea2

�, (1)

x2, f1, f2 – inputs

x1, Rea2 (reaction forces) – outputs

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 37

/ 49

Page 38: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

General constraints vs. frontal solver on subdomain

central idea – split matrix C according to types of constraints

corners as Dirichlet boundary conditions, i.e. fixed variables

averages enforced by Lagrange multipliers – matrix Cf

coarse problem construction on subdomain (index i omitted)

Aff Afc CT

fAcf Acc 0Cf 0 0

Ψc

f Ψavgf

I 0λc λavg

=

0 0rea rea0 I

.

+ get rid of indefiniteness by eliminating the first block

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 38

/ 49

Page 39: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Preconditioner setup

1 Forward step of frontal solver with corners marked as fixed variablesin matrix A.

2 Find A−1ff CT

f by backward solve by frontal solver�

Aff Afc

Acf Acc

� �A−1ff CT

f0

�=

�CTf0

�+

�0�

Cf A−1ff Afc

�T

�.

3 Construct Cf A−1ff CT

f and factor it by LAPACK.4 Backward solve of dual problem by LAPACK for λ from

Cf A−1ff CT

f λ = −�Cf A

−1ff Afc I

�.

5 Backward solve for Ψf by frontal solver�

Aff Afc

Acf Acc

� �Ψc

f Ψavgf

I 0

�=

�−CT

f λ0

�+

�0

Rea

�.

6 Compute local AC = ΨTAΨ = ΨT

�−CT

f λRea

�.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 39

/ 49

Page 40: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

General constraints vs. frontal solver

subdomain problem solution

Aff Afc CT

fAcf Acc 0Cf 0 0

uf0µ

=

rrea0

.

+ get rid of indefiniteness by eliminating the first block

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 40

/ 49

Page 41: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Algorithm of preconditioning action on subdomain

1 Backward step of frontal solver for A−1ff r

�Aff Afc

Acf Acc

� �A−1ff r0

�=

�r0

�+

�0

Rea

�.

2 Backward step of LAPACK for µ

Cf A−1ff CT

f µ = Cf A−1ff r .

3 Backward step of frontal solver for uf�

Aff Afc

Acf Acc

� �uf0

�=

�−CT

f µ+ r0

�+

�0

Rea

�.

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 41

/ 49

Page 42: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Implementation

subdomain problems - frontal solver + LAPACK

coarse problem - MUltifrontal Massively Parallel sparse direct Solver(MUMPS) http://mumps.enseeiht.fr

mainly Fortran 77 programming language, partly Fortran 90, MPIlibrary

tested onSGI Altix 4700, CTU, Prague, CR72 processors Intel Itanium 2, OS Linux

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 42

/ 49

Page 43: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement

33 186 quadratic elements, 544 734 unknowns

16 subdomains, 35 corners, 12 edges, and 35 faces

32 subdomains, 57 corners, 12 edges, and 66 faces

16 processors of SGI Altix 4700

Decomposition into 32 subdomains

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 43

/ 49

Page 44: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement

von Mises stresses in improved design

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 44

/ 49

Page 45: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement

0

20

40

60

80

100

0 1000 2000 3000 4000

cond

ition

num

ber e

stim

ate

[/]

number of corners [/]

Condition number SGI Altix 470016 processors

nsub = 16nsub = 32

Condition number for adding corners(Leipzig) Adaptive Multilevel BDDC

Max-Planck Institute, Leipzig, March 19, 2010 45/ 49

Page 46: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement

0

200

400

0 1000 2000 3000 4000

wal

l tim

e [s

econ

ds]

number of corners [/]

Wall times for variable coarse problem SGI Altix 470016 processors

nsub = 16nsub = 32

Wall clock time for adding corners(Leipzig) Adaptive Multilevel BDDC

Max-Planck Institute, Leipzig, March 19, 2010 46/ 49

Page 47: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement, 16 subdomains

coarse problem C. C.+E. C.+F. C.+E.+F.

iterations 35 34 26 26cond. number est. 96 96 65 65

factorization (sec) 91 80 78 106pcg iter (sec) 53 49 38 37

total (sec) 183 166 153 181

adding averages to 335 corners

coarse degrees of freedom:

C. - Corners only

C.+E. - Corners and averages on Edges

C.+F. - Corners and averages on Faces

C.+E.+F. - Corners and averages on Edges and Faces

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 47

/ 49

Page 48: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Hip joint replacement, 32 subdomains

coarse problem C. C.+E. C.+F. C.+E.+F.

iterations 35 32 30 27cond. number est. 149 70 59 46

factorization (sec) 60 57 59 62pcg iter (sec) 49 40 37 34

total (sec) 128 115 113 113

adding averages to 557 corners

coarse degrees of freedom:

C. - Corners only

C.+E. - Corners and averages on Edges

C.+F. - Corners and averages on Faces

C.+E.+F. - Corners and averages on Edges and Faces

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 48

/ 49

Page 49: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 3D (data and visualization: J. Sıstek)

dofs: 107 811, substructures: 8, corners/edges/faces: 30/16/15

cube: E = 106 Pa, ν = 0.45, bars: E = 2.1 · 1011 Pa, ν = 0.3.

(CCM, 2010) Adaptive – Multilevel BDDC Denver, : 25 / 56

Page 50: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Numerical results in 3D: Composite cube

dofs: 107 811, substructures: 8, corners/edges/faces: 30/16/15

constraint Nc κ it

c 90 408114 455

c+e 138 125378 307

c+e+f 183 18915.1 211

c+e+f (3eigv) 183 1267.61 81

Adaptive constraints:τ Nc �ω κ it

∞(=c+e) 138 268390.00 125378.00 307

10000 148 5096.07 1843.70 104

1000 159 368.78 173.57 38

5 198 4.99 4.55 21

2 465 <2 2.80 16

The local indicators are in the interval�3.9, 2.7 · 105

�.

(CCM, 2010) Adaptive – Multilevel BDDC Denver, : 27 / 56

Page 51: Adaptive Multilevel BDDCmath.ucdenver.edu/~jmandel/slides/leipzig.pdf · Max-Planck Institute, Leipzig, March 19, 2010 3 / 49. Basic substructuring assemble elements into substructures,

Conclusion

distinguish between point constraints and averages for frontal solver

many matrices needed in BDDC are just a simple side-product of thefrontal solver (reactions)

‘minimal’ number of corners does not assure minimal solution time

constraits on edges and/or faces can considerably shorten the solutiontime

more sophisticated (adaptive) way for selection of constraints -ongoing research

(Leipzig) Adaptive Multilevel BDDCMax-Planck Institute, Leipzig, March 19, 2010 49

/ 49