Block selective algebraic multigrid in...

ESI 5th

OpenFOAMUser

Conference

Tessa Uroic,Hrvoje Jasak

Introduction:pressure–velocitycouplingPerformance of theimplicitly coupledalgorithm

SAMG forimplicitlycoupledsystemThe idea of multigrid

The basic multigridalgorithm

Interpolation

Coarse matrixformation - tripleproduct

SAMG forblock matrices

ResultsBackward–facingstep

Engine cooling

Bluff body

Summary

Block–selective algebraic multigrid infoam-extend

Tessa Uroic, Hrvoje Jasak

Chair of TurbomachineryDepartment of Energy, Power Engineering and Environment

Faculty of Mechanical Engineering and Naval Architecture, University ofZagreb

[email protected], [email protected]

October 16, 2017

1 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

OverviewFSB

1 Introduction: pressure–velocity couplingPerformance of the implicitly coupled algorithm

2 SAMG for implicitly coupled systemThe idea of multigridThe basic multigrid algorithmInterpolationCoarse matrix formation - triple product

3 SAMG for block matrices

4 ResultsBackward–facing stepEngine coolingBluff body

5 Summary

2 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Pressure–velocity couplingFSB

Steady state, incompressible, single-phase, turbulent Navier-Stokes equations:

∇•u = 0, (1)

∂u∂t

+∇•(uu)−∇• (ν∇u) = −∇p. (2)

The connection between velocity and pressure is linear, but there is a non–linearconvection term in the momentum equation which is linearised by using the old oravailable values of the flux:

∇•(uu) = ∇•(uoun). (3)

Final form of the implicitly coupled pressure–velocity system:[[Au] [∇(.)]

[∇•(.)] [∇•(D−1u ∇(.))]

][up

]=[

0∇•(D−1

u ∇(po))

], (4)

3 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Pressure–velocity couplingFSB

Spatial discretisation in the framework of Finite Volume method produces sparsematrices - first level :

[a1,1] [a1,2] · · · [a1,N ][a2,1] [a2,2] · · · [a2,N ]

......

. . ....

[aN,1] [aN,2] · · · [aN,N ]

[φ1][φ2]

...[φN ]

=

[b1][b2]

...[bN ]

, (5)

• [aij ] is a block–coefficient denoting the coupling between cells i and j,• [φi] is the vector containing the unknowns’ field value solved for in the cell

i,• [bi] is the right hand side term for cell i.

Block–system is implemented as point–ordered, meaning that the unknown vectorarray is organised cell by cell, rather than variable by variable.

4 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Pressure–velocity couplingFSBIn the block–system, each unknown [φi] belonging to cell i is a vector:

φi =

uxi

uyi

uzi

pi

, (6)

Each block–matrix coefficient [aij ] is a matrix describing the connection betweenvelocity components and pressure:

ai,j =

auxi,uxj auxi,uyj auxi,uzj auxi,pj

auyi,uxj auyi,uyj auyi,uzj auyi,pj

auzi,uxj auzi,uyj auzi,uzj auzi,pj

api,uxj api,uyj api,uzj api,pj

. (7)

In Eqn. (7), block–matrix coefficient [aij ] is divided into four parts:• the upper 3× 3 block denotes the coupling between velocity components.

Off-diagonal entries are zero when using the old values of the flux.• the upper 3× 1 block denotes the pressure–velocity coupling in the

momentum equation.• the lower 1× 3 block denotes the pressure–velocity coupling in the pressure

equation.• the lower 1× 1 block denotes the pressure equation contribution.

5 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Performance of the implicitly coupledalgorithmFSB

• The coupled algorithm solves the velocityand pressure equation simultaneously in asingle matrix.

• Matrix is 16x larger than the matrices insegregated algorithms (example: SIMPLE,PISO). More storage is needed!

• The block–matrix contains equations ofdifferent types: hyperbolic (velocity) andelliptic (pressure), meaning that the choiceof optimal linear solver has become moredifficult.

• Underrelaxation is neccesary only for themomentum equation due to thenon–linearities in the convection term.Pressure equation does not needunderrelaxation.

• The system converges in far less iterationsthan the segregated counterparts, but asingle iteration is more expensive!

• In combination with implicitly coupledturbulence models, implicitly coupled solverprovides better stability, robustness andreduced time–to–solution than segregatedalgorithms.

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 5 10 15 20 25 30 35

Res

idu

al

Execution Time [s]

backwardFacingStep

Ux

Uy

p

Ux (simpleFoam)

Uy (simpleFoam)

p (simpleFoam)

6 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

The idea of multigridFSB

• Fixed–point iterative solvers can quickly reduce (local) high–frequencyerrors and when the (global) low–frequency errors remain, the convergencestalls.

• It was noticed that the (global) low–frequency errors are removed at a rateinversely proportional to the size of the computational mesh.

• Multigrid algorithms help reduce the low–frequency error by successivelysolving the system on coarser meshes.

• The low–frequency errors on a fine mesh become high–frequency errors ona coarse mesh and can be efficiently eliminated.

Figure 1: Effects of error smoothing and restriction. (by Robert Falgout)

7 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

The basic multigrid algorithmFSB

1 On the fine level, use one of the fixed–point methods (Jacobi,Gauss–Seidel, ...) to calculate the approximate solution of the system:Ax = b.

2 Calculate the residual:rF = bF −AxF .

3 On the coarse level, use a fixed–point method to solve:ACeC = rC .

4 Use the error calculated on the coarse level to correct the solution on thefine level:xnew = xF + eF

How to define operators for transferring the residual and correction from fine levelto coarse level, and vice versa? There are multiple possibilities, some of which willbe explained in the following section. The terminology and theoreticalbackground are based on the work of Klaus Stuben and Tanja Clees.Please note, algebraic multigrid operates only on matrix level and does not needinformation from the computational mesh.

8 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary


• Options for forming interpolation by injection or weighted residual method:• Aggregative multigrid - AAMG: equations are grouped into clusters

(multiple fine cells are joined to form a single coarse cell). The groups are

formed based on the size of off-diagonal coefficients:a2

ij

aiiajjshould exceed

a certain size. The interpolation operator is piecewise constant (injection).The coarse level matrix coefficients are obtained as a sum of fine levelmatrix coefficients. This method was reported as the most efficient for theimplicitly–coupled pressure velocity system by several authors.

• Selective multigrid - SAMG: The set of coarse equations is a subset of finelevel equations. The choice of the coarse equations is based on the numberof equations which depend on a single equation. The interpolation operatoris obtained from approximation of the algebraically smooth error. Coarselevel matrix is obtained by a triple matrix product of restriction, fine levelmatrix and prolongation operator.

9 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary


Figure 2: Visual comparison of the two algebraic multigrid principles.

10 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

CoarseningFSB

Figure 3: Example: Mesh consisting of 9 cells with the corresponding matrix.Three equations were chosen as coarse: 0, 6, 7. All fine equations will beinterpolated from the coarse equations.

a00 a01 a02a10 a11 a13 a14a20 a22 a24 a25

a31 a33 a36a41 a42 a44 a46 a47

a52 a55 a57a63 a64 a66 a68

a74 a75 a77 a78a86 a87 a88

11 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

InterpolationFSB

• The interpolation for transferring the solution correction from coarse to finemesh is calculated explicitly from the fine level matrix.

• Interpolation weights for coarse equations are equal to 1 (injection).• Interpolation weights for fine equations are calculated using an

approximation of the equation which describes the algebraically smootherror:

aiiei +∑j∈Ni

aijej = 0. (8)

aiiei = −∑j∈Ci

aijej −∑j∈Fs

i

aijej −∑j∈Fw

i

aijej , (9)

• where:• Ni is the subset of all neighbours of i• Ci is the subset of strong coarse neighbours of i• Fw

i is the subset of strong fine neighbours of i• Fs

i is the subset of weak fine neighbours of i

12 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

InterpolationFSB• Treatment of fine connections is crucial for the definition of interpolation

weights:• Weak connections are collapsed (added to the diagonal).• Strong fine connections cannot be ignored and added to the diagonal.

They can either be added to strong coarse subset via the scalingfactor (direct interpolation) or their contribution may be added usingthe coarse equations from which they are interpolated (standardinterpolation).

• Using the principles of the direct interpolation, interpolation weight forcontribution from a single coarse equation to the fine equation is calculatedas:

w = −α1

aii +∑

j∈Fwi

aij

aCij , (10)

• where α is the scaling factor which takes into account the eliminated strongfine connections:

α =

∑j∈Ci

aij +∑

j∈Fsi

aij∑j∈Ci

aij

> 1 (11)

13 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Coarse matrix formation – triple productFSB

• Coarse level matrix is formed by multiplyingthe restriction matrix, fine level matrix andprolongation matrix: AC = R ·AF · P

• If the interpolation is defined by using onlydirect connections (first neighbours), coarselevel communication can be establishedthrough maximally two fine equations.

• This means that two cells (equations) whichwere chosen as coarse on the fine level, willshare a face (off-diagonal coefficient) on thecoarse level if there are two (or less) fine cellsbetween them.

• This principle for calculating the coarse leveloperator leads to increase in communication:matrices become denser, with moreoff–diagonal coefficients. Thus: coarse levelmatrices represent a fictive polyhedral mesh.

• Problem: Gauss–Seidel and ILU0 are nolonger efficient smoothers(diagonal–dominance issues and densermatrices).

• Solution: Implementation of ILUC0smoother.

14 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

SAMG for block matricesFSB• Point-based SAMG: unknowns are discretised on the same mesh, thus the

hieararchy of levels is the same for all variables.• Coarsening is performed on the primary matrix which should reflect the

couplings in a reasonable sense.• Important questions:

• Is it important to take variable cross-couplings into account. No.• If each unknown should be interpolated only from variables

corresponding to the same unknown, do we need to consider eachunknown separately or is it sufficient to use the same formulae for allunknowns? Yes, it is sufficient.

• The answers to these questions lead to a single-unknown interpolation:interpolation formulae are computed variable-wise and are identical for thevariables belonging to the same cell.

• Thus, choose a primary matrix P which represents the couplings betweenthe equations, i.e. calculate a norm of the block–coefficient:

∣∣∣∣∣∣∣auxi,uxj auxi,uyj auxi,uzj auxi,pj

auyi,uxj auyi,uyj auyi,uzj auyi,pj

auzi,uxj auzi,uyj auzi,uzj auzi,pj

api,uxj api,uyj api,uzj api,pj

∣∣∣∣∣∣∣pressure norm

→ aprimaryi,j = api,pj .

15 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Backward–facing step: CoarseningFSB

Figure 4: First level of coarsening: AAMG (2394 equations, 9494 off–diagonalcoefficients), SAMG (2400 equations, 17530 off–diagonal coefficients).

Figure 5: Third level of coarsening: AAMG (561 equations, 2382 off–diagonalcoefficients), SAMG (231 equations, 2682 off–diagonal coefficients). 16 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Backward–facing step: CoarseningFSB

Figure 6: Last level of coarsening: AAMG (6 equations, 14 off–diagonalcoefficients), SAMG (6 equations, 18 off–diagonal coefficients).

SAMG AAMG

Coarse level Equations Off–diag Average Equations Off–diag Average

1 2400 17 530 7.3 2394 9494 4.02 810 7 800 9.6 1189 4872 4.13 231 2682 11.6 561 2382 4.24 64 630 9.8 268 1150 4.35 19 114 6 126 550 4.46 6 18 3 56 234 4.27 - - - 26 90 3.58 - - - 12 34 2.89 - - - 6 14 2.3

17 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Backward–facing step: ConvergenceFSB

Solver Iteration Time [s] Lin. iterations

SAMG (V-cycle) 211 8.6 388SAMG (W-cycle) 211 10.8 348AAMG 214 13.7 2381BiCGStab 219 8.0 6026GMRES 218 24.9 6950Segregated 500 32.0 /

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 5 10 15 20 25

Res

idu

al

Execution Time [s]

backwardFacingStep

Ux (SAMG-V)

Uy (SAMG-V)

p (SAMG-V)

Ux (SAMG-W)

Uy (SAMG-W)

p (SAMG-W)

Ux (AAMG)

Uy (AAMG)

p (AAMG)

Ux (BiCGStab)

Uy (BiCGStab)

p (BiCGStab)

Ux (GMRES)

Uy (GMRES)

p (GMRES)

18 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Engine coolingFSB

Figure 7: Engine cooling case, unstructured mesh, 156 739 cells

19 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Engine cooling: ConvergenceFSB

Solver Iteration Time [s] Lin. iterations

SAMG (V-cycle) 70 284.6 197SAMG (W-cycle) 70 403.1 194AAMG 120 4007.9 12 006BiCGStab 80 547.7 6365GMRES 140 3387 13 800Segregated 300 502.5 /

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 500 1000 1500 2000 2500 3000 3500 4000

Res

idu

al

Execution Time [s]

Engine Cooling

Ux (SAMG-V)

Uy (SAMG-V)

Uz (SAMG-V)

p (SAMG-V)

Ux (SAMG-W)

Uy (SAMG-W)

Uz (SAMG-W)

p (SAMG-W)

Ux (AAMG)

Uy (AAMG)

Uz (AAMG)

p (AAMG)

Ux (BiCGStab)

Uy (BiCGStab)

Uz (BiCGStab)

p (BiCGStab)

Ux (GMRES)

Uy (GMRES)

Uz (GMRES)

p (GMRES)

20 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Engine cooling: ConvergenceFSB

1e-09

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 10 20 30 40 50 60 70 80 90 100

Res

idu

al

Linear Solver Iteration

Engine Cooling

Ux (SAMG)

Uy (SAMG)

Uz (SAMG)

p (SAMG)

Ux (AAMG)

Uy (AAMG)

Uz (AAMG)

p (AAMG)

Ux (BiCGStab)

Uy (BiCGStab)

Uz (BiCGStab)

p (BiCGStab)

Ux (SAMG-twoNorm)

Uy (SAMG-twoNorm)

Uz (SAMG-twoNorm)

p (SAMG-twoNorm)

Figure 8: Comparison of convergence of linear solvers in a single iteration of thenon–linear pressure–velocity solver.

21 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Bluff bodyFSB

A more complex case: aerodynamics test case, a bluff body with an upswept backsection. When placed near the ground, the back section forms a diffuser. Themesh is unstructured, 9 269 072 cells of different types (hexahedral, tetrahedral,polyhedral).Implicitly coupled pressure–velocity solver with SAMG (V–cycle) for the linearmatrix converges to the prescribed tolerance with no oscillations (10−5).

22 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

Bluff bodyFSB

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 500 1000 1500 2000 2500

Resid

ual

Iteration

Ux (simpleFoam)

Uy (simpleFoam)

Uz (simpleFoam)

p (simpleFoam)

BiCGStab Ux

BiCGStab Uy

BiCGStab Uz

BiCGStab p

SAMG Ux

SAMG Uy

SAMG Uz

SAMG p

1e-08

1e-07

1e-06

1e-05

1e-04

1e-03

1e-02

1e-01

1e+00

0 20000 40000 60000 80000 100000 120000

Resi

du

al

Execution time [s]

Ux (simpleFoam)

Uy (simpleFoam)

Uz (simpleFoam)

p (simpleFoam)

BiCGStab Ux

BiCGStab Uy

BiCGStab Uz

BiCGStab p

SAMG Ux

SAMG Uy

SAMG Uz

SAMG p

23 / 24

ESI 5th

OpenFOAMUser

Conference





Interpolation




Engine cooling

Bluff body

Summary

SummaryFSB

• We have implemented the selective algebraic multigrid algorithm (SAMG)for block matrices.

• The algorithm follows the principles of the classical AMG (Stuben), but usesa primary matrix (Clees) for coarsening and calculation of interpolation.

• Coarsening procedure in both AAMG and SAMG is based on the strengthof connection, but coarse level operators are different.

• Coarse level matrices in SAMG have more off–diagonal coefficients per rowthan the fine level matrix. The existing implementation of ILU0 smootherwas not efficient on these matrices and we implemented a new smootherbased on Crout’s lower–upper factorisation.

• The algorithm dramatically outperforms AAMG and Krylov subspace solverin terms of convergence per number of iterations, but is more expensive fora single iteration, due to the high number of matrix–vector multiplications(restriction of residual, prolongation of correction, triple product).

24 / 24

Block selective algebraic multigrid in...

Documents

Transcript of Block selective algebraic multigrid in...