Spectral Elements for Two-Dimensional Resistive MHDhientzsc/losalamos.pdf · 2006-07-03 ·...

Spectral Elements for Two-Dimensional

Resistive MHD

Bernhard HientzschCourant Institute of Mathematical Sciences

New York Universitymailto:[email protected]

http://www.math.nyu.edu/~hientzsc

May 25, 2005

Numerical Analysis SeminarLos Alamos National Laboratory

Los Alamos, New Mexico

Working with H.R. Strauss at NYU.

Bernhard Hientzsch SEM for 2D resistive MHD

Overview

• Why spectral elements: the approach

• Incompressible MHD in 2D: in primitive variables and potentials

• Time discretization (FD), space discretization (SEM), complete algorithm

• Tilting mode examples. Setup (IV/BV). Variables, energies, peakcurrents, growth rates

• C1 continuous elements and mapped elements

• Implementation in C/LAPACK/ATLAS resp. SUNPERFLIB

• Numerical observations and future work

Courant Institute, New York University 1


Why spectral elements: the approach

• Exponential convergence for (standard) problems with (piecewise)smooth solutions. Alignment, postprocessing, filtering etc. for otherproblems and solutions might restore exponential convergence.

• Fast application of stiffness and mass matrix. Fast solvers for rectangularelements and rectangles. Helmholtz: generalized Sylvester equation.

• Relatively straightforward implementation. Expressed in matrix-matrixmultiplications, element-wise multiplications and a few other matrixand vector operations. Translates into matrix-heavy MATLAB orC/C++/FORTRAN using LAPACK/BLAS/...

• Can run at high percentage of peak at modern computer architectures.(Sparse block matrix with small dense blocks.)



General philosophy

• Try out spectral element type discretizations on more complex problems.(Before: linear elliptic, Maxwell, and some nonlinear elliptic equations.)

• Rapid prototype the discretizations and methods. Choose simple andstraightforward approaches first to get into the problem as fast as possible

• Rapid prototype the programming. MATLAB/PYTHON 7→C/LAPACK/BLAS 7→ PETSC 7→ M3D.

• Try first on structured grids and easy domains. Fast solvers availablefor those. Quick turnaround and extensive testing possible in MATLABwith those solvers an domains. Validate and debug more general solversthere. In general, can use version in MATLAB to verify, validate anddebug codes in C/..., at least on small examples.



Description of the project - relation to M3D

• The formulation(s) and time discretization is given and fixed by M3D (orcoding it should require only small changes), but there were/are differentversions with different space discretizations. All of them until now lowerorder, mostly even just linear elements.

• My project is to test spectral elements (SEM) as a space discretizationfor this formulation on some examples; validate them as an approach onsome examples; and propose algorithms and write modules for inclusionin M3D.

• Also, I am acting as somewhat of a consultant for some issues in finiteelements/numerical analysis/scientific computing for the M3D group.Some of my suggestions have made their way into the code ...



Incompressible resistive MHD: primitive variables

∇u is gradient ∇u :=(

∂u∂x, ∂u

∂y

)

∇2u is Laplacian ∇2u := ∂2u∂x2 + ∂2u

∂y2

B: magnetic field.v: velocity field.ρ: density, here assumed constant. µ: viscosity. η: resistivity.

∂B

∂t= curl(v × B) + η∇2B (1)

ρ∂v

∂t= −ρv · ∇v + curlB ×B + ρµ∇2v (2)

∇ · v = 0 (3)

∇ · B = 0 (4)



Incompressible MHD: potential form in 2D

(vorticity-flux)

[a, b] := ∂a∂x

∂b∂y

− ∂b∂x

∂a∂y

. Poisson bracket.

v = curlφ with velocity flux φ. B = curlψ with magnetic flux ψ.

Ω: vorticity. C: current density.

∂Ω

∂t= [C,ψ] − [Ω, φ] + µ∇2Ω (5)

∂ψ

∂t= −[ψ, φ] + η∇2ψ (6)

∇2φ = Ω (7)

C = ∇2ψ (8)



Incompressible MHD: potential form in 2D

(vorticity-current)

∂Ω

∂t= [C,ψ] − [Ω, φ] + µ∇2Ω (9)

∂C

∂t= [φ,C] + 2

[

∂φ

∂x,∂ψ

∂x

]

+ 2

[

∂φ

∂y,∂ψ

∂y

]

+ [Ω, ψ] + η∇2C (10)

∇2φ = Ω (11)

∇2ψ = C (12)



Time discretization (vorticity-flux, semi-implicit)

Diffusive terms implicit, all others explicit. Leapfrog, i.e., use mostcurrent variables. Vorticity-current is discretized in the same way, just morebrackets.

Ωn+1 − Ωn

∆t= [Cn, ψn] − [Ωn, φn] + µ∇2Ωn+1 (13)

∇2φn+1 = Ωn+1 (14)

ψn+1 − ψn

∆t= −[ψn, φn+1] + η∇2ψn+1 (15)

Cn+1 = ∇2ψn+1 (16)



PDEs to be solved in each time step (vorticity-flux)

Ωn+1 − µ∆t∇2Ωn+1 = Ωn + ∆t [Cn, ψn] − [Ωn, φn] (17)

∇2φn+1 = Ωn+1 (18)

ψn+1 − η∆t∇2ψn+1 = ψn − ∆t[ψn, φn+1] (19)

Cn+1 = ∇2ψn+1 (20)

(17) and (19) are Helmholtz solves, for (I + α∇2) with α = −µ∆t andα = −η∆t, respectively. For µ = 0 or η = 0, respectively, Helmholtz solvessimplify to direct formulae or L2 projection for the new values taking intoaccount possible boundary conditions. (18) and (20) are standard Laplacesolves resp. applications of Laplace operator.



Time stepping algorithm

A time-stepping algorithm can be implemented like

Ωn+1 = HHSolve(−µ∆t,Ωn + ∆tP (Cn, ψn) − ∆tP (Ωn, φn))

φn+1 = LapSolve(Ωn+1)

ψn+1 = HHSolve(−η∆t, ψn − ∆tP (ψn, φn+1))

Cn+1 = ApplyLap(ψn+1)

Some optimization possible in the computation of the right hand sideby saving terms occuring at several places. Right hand side assembly needsonly a module for Poisson bracket terms if modularity is more important.Different BC/DOF/continuity only influence HHSolve, LapSolve, ApplyLap,and P, can reuse the same frame.



Time-stepping code fragment in MATLAB

omexrhs=pbrh(c,psi,disc)+pbrh(phi,omega,disc)+soOm;

omrhs=omega+dt*omexrhs;

omega=SolveHH(omrhs,disc,hhdisco);

phi=SolveSLap(omega,disc);

psrhs=dt*(pbrh(phi,psi,disc)+soPs-eta*ApplyLap0Bc(psi,disc));

dpsi=SolveHH(psrhs,disc,hhdiscp);

psi=psi+dpsi;

c=ApplySLap0Bc(psi,disc);

Intermediate results are put into variables because of other optionsand algorithmic possibilities not shown here. pbrh is a function handle.disc is a context structure for the discretization containing one-dimensionalmatrices etc. hhdisco and hhdiscp are context structures for the fastHelmholtz solves. All solves assume zero Dirichlet data. BCs from tiltingmode problem.



Spectral elements, Introduction

• Approximate function in elements by high-order polynomials from tensorproduct space. Parametrize by values on (mapped) GLL grid.

• Differentiation, interpolation between grids, exact and approximateintegrals of products 7→ matrices acting on values on grid.

• On multi-dimensional non-distorted rectangular elements, matrices aretensor products matrices. F.i., x-derivative in 2D in element aligned withaxes is (D ⊗ I): derivative only acts along one coordinate.

• Bilinearly or isoparametrically mapped elements: operators are productsof block tensor product matrices and diagonal matrices.

• (A⊗B)u can be rewritten as matrix-matrix multiplication AUBT , whereU and result are 2D arrays instead of vectors.



Spectral elements, Helmholtz

• PDE: u + α∇2u = f , integrate by parts to obtain weak form: ∀v ∈H1

0 find u ∈ H10 satisfying (u, v) − α(∇u,∇v) = (u, v) − α(ux, vx) −

α(uy, vy) = (f, v).

• Instead of H10 , use polynomial space and matrices: vTMu −

αvTDTxMDxu− αvTDT

y MDyu = vTMf .

• (Mx⊗My)u−α(Dx,T ⊗ I)(Mx⊗My)(Dx⊗ I)u−α(I⊗Dy,T )(Mx⊗My)(I ⊗Dy)u = (Mx ⊗My)f

• (Mx ⊗My − αKx ⊗My − αMx ⊗Ky)u = (Mx ⊗My)f

• ((12M

x − αKx) ⊗My +Mx ⊗ (12M

y − αKy))u = (Mx ⊗My)f

• (Ax ⊗My +Mx ⊗ Ay)u = (Mx ⊗My)f or AxUMy,T +MxUAy,t =MxFMy,T (Generalized Sylvester equation.)



Spectral elements, Fast Solvers

Several methods to solve generalized Sylvester equations:

• Solve two generalized eigenvalue problems and diagonalize system bywriting in eigenbasis (fast diagonalization method).

• Transform matrix pairs to Schur form, and the transformed system(triangular in a sense) is solvable by back-substitution.

• Transform matrix pair to Schur/Hessenberg form, and then solvetriangular and Hessenberg systems.

First choice seems to be fast and stable enough for our computations.Second and third choice should be more stable, have implementation ofsecond choice. Do not see much difference in numerical behavior.



Poisson brackets: different analytical expressions

Same continuous object/operator if functions are smooth enough,but different expression. These different expressions lead to differentdiscretizations with different properties.

[φ,ψ] := φxψy − φyψx

div(ψz ×∇φ) = div(−ψφy, ψφx, 0) = [φ,ψ] = −[ψ, φ]

[φ,ψ] = div(ψz ×∇φ) = − div(ψ curlφ) = − div(φ curlψ)

= div(ψz ×∇φ) = − div(φz ×∇ψ)

= curl(φz) · ∇ψ = curl(ψz) · ∇φ



Poisson brackets: discretization INeed to approximate ([a, b], c) =

(

∂a∂x

∂b∂y

− ∂a∂y

∂b∂x, v

)

In general [a, b] is a nonlinear term, since both a and b are usuallydependent variables in the system. [a, b] could be computed exactly as apolynomial, but with much higher degree, if the nonlinearity is only of finiteorder. Afterwards, that needs to be projected/filtered back.

First choose the analytical expression to start with. Possible choices are:

• Definition of Poisson bracket: ∂a∂x

∂b∂y

− ∂a∂y

∂b∂x

• Use divergence form and integrate by parts (Glasser), for instance[a, b], c = (div(bz × grad a), c) = −(bz ×∇q,∇c) + boundary terms

• curl-grad form - possible alternative, but don’t know of good reason touse it.



Using Poisson brackets: Implementation

Procedure:

• Given a and b, approximate [a, b] on some grid/polynomial space. Canfilter it there.

• Integrate by appropriate quadrature rule (after interpolating [a, b] to itwith appropriate spectral element interpolation). This also maps [a, b] toits action on a lower-degree space, and therefore truncates/projects [a, b]to a lower-order approximation.

• Filter the Poisson bracket, or

• Filter the entire right hand side at once.



Using Poisson brackets: GLL implementationIf the different representations are all GLL representations, [a, b] is

approximated appropriately, and integration happens on the GLL grids, thediscretization of ([a, b], c) can be written in terms of matrices and opertions

([a, b], v) ≈ vT In1,Tn Mn1Fn2

n1((DxI

n2n a) ~ (DyI

n2n b) − (DyI

n22 a) ~ (DxI

n2n b))

Experimenting with different filters such as L2 projections, aninterpolatory filter proposed by Paul Fischer, spectral filtering and diagnostic(transforming to spectral coefficient space for the modal expansion) and anL2 projection into a C1 continuous global space.

The last seems to work best for the problems up to now.

This is ongoing work, looking at the different polution effects andnumerical instabilities that occured in my computer experiments and thoseof others.



Tilting mode problem - Setup

Introduce polar coordinates x = r cos θ, y = r sin θ and use separableform in polar coordinates,

ψ(t = 0) = ψ0,rad(r) cos θ Ω(t = 0) = εΩ0,pert(r)

with the radial functions (k being the first positive zero of J1, k ≈ 3.8317)

ψ0,rad(r) =

2J1(kr)kJ0(k) for r ≤ 1r2

−1r

for r ≥ 1Ω0,pert(r) = 4(r2 − 1) exp(−r2)

The following boundary conditions were used in this problem:

φ = 0 C = 0∂ψ

∂t= 0 Ω = 0



Algorithmic choices for initial examples.

• MATLAB: time-integration on one element or on a rectangular array:explicit, semi-implicit, ODE suite. C: semi-implicit.

• Sometime scaled version of problem to compute only on [−1, 1]2.

• Split rectangle into M ×M elements of degree N ×N .

• Use the tensor product structure of Helmholtz and Laplaceequations for fast solvers. (Fast diagonalization methods/Blockdiagonalization methods/Hessenberg-Schur methods for generalizedSylvester equations.)

• Typical: µ = 0.005, η = 0 or η = 0.005, ε = 0.0001 or ε = 0.001,integration up to time t = 2, 4, 10



50 × 50 elements of degree 2 × 2

PPE = 10, ∆t = 0.001, tfinal = 4.0.




PPE = 10, ∆t = 0.001, tfinal = 4.0.



10 × 10 elements: degrees 5 × 5, 6 × 6

PPE = 10, ∆t = 0.001, tfinal = 4.0.



5 × 5 elements: Kinetic energiesUpper left: degree 5, upper right: degree 6, lower left: degree 10, lower

right: degree 20.

-20

-15

-10

-5

0 0.5 1 1.5 2 2.5 3 3.5 4

kine

tic e

nerg

y

x

kinetic energy (data from 5x5deg5.enercur.dat)

log(kinetic energy)

-20

-15

-10

-5

0 0.5 1 1.5 2 2.5 3 3.5 4

kine

tic e

nerg

y

x


log(kinetic energy)

-20

-15

-10

-5

0 0.5 1 1.5 2 2.5 3 3.5 4

kine

tic e

nerg

y

x


log(kinetic energy)

-20

-15

-10

-5

0 0.5 1 1.5 2 2.5 3 3.5 4

kine

tic e

nerg

y

x


log(kinetic energy)



5 × 5 elements: Growth rates (+elapsed time)

4000 time steps. MATLAB on Sun Blade workstation.

Degree Estimated growth rate Elapsed time

5 1.2065 243.3s6 1.2543 265.4s10 1.2398 494.4s20 1.2417 8843s



10 × 10 elements of degree 5, Poisson brackets, Ω

At time t = 6.0. Upper row: Averaged continuous derivative. Lowerrow: piecewise continuous derivative. Left column: Integrated on samegrid, i.e., under-integration. Right column: Integrated exactly on finer grid.



10 × 10 elements of degree 5, Poisson brackets, Ψ



10 × 10 elements of degree 5, Poisson brackets, Φ



10 × 10 elements of degree 5, Poisson brackets, C



5 × 5 elements of degree 10, no filtering

Time t = 8.0. Upper row: Averaged continuous derivative. Lower row:piecewise continuous derivative. Left column: Integrated on same grid, i.e.,under-integration. Right column: Integrated exactly on finer grid.



5 × 5 elements of degree 10, C1 filteringTime t = 8.0. Upper row: Averaged continuous derivative. Lower row:

piecewise continuous derivative. Left column: Integrated on same grid, i.e.,under-integration. Right column: Integrated exactly on finer grid.



C1 continuous elementsIdea: most edge effects and pollution are concentrated close to the

boundary and seem to be associated to jumps in the derivative, which areallowed for discretizations with C0 finite elements.

Such jumps cannot occur if we enforce C1 continuity. The problem,however, might just manifest itself differently in such an approach. Thereforewe designed and implemented a spectral element with C1 continuity.

For this, we need

• appropriate definitions of the degrees of freedom

• a layout or ordering of the degrees of freedom into a array, and,

• if possible, fast solvers and fast matrix-vector multiplication.



C1 continuous elements - Degrees of freedom, geometry

Function value: filled circles. Normal derivatives: horizontal and verticalarrows. Mixed second derivatives: diagonal arrows. (Spectral generalizationof BFS: Bogner-Fox-Smith.)

On the right: Reparametrize in coordinate derivatives, horizontal arrows:ux, vertical arrows: uy, diagonal arrows: uxy.



C1 continuous elements - Degrees of freedom, arrays

The degrees of freedom just shown fit into an array (left), and the arraycan be written as a tensor product representation of two one-dimensionalrepresentations (right). One element is shown on both sides, and the innerrectangle on the left surrounds the interior degrees of freedom.



C1 continuous elements - computational aspects

• We work on rectangles or mapped rectangles. For rectangles, use thedegrees shown. Start with one-dimensional degrees of freedom. Takingtensor products of these gives us degrees of freedom involving ux, uy onthe edges, and uxy in the corners.

• Easy transformation to normal spectral element degrees of freedom oneach element. Can use the same machinery. For rectangle split intoregular array of elements, subassembled form for Laplace and Helmholtzequation is still generalized Sylvester equation. Same fast solver.

• On mapped elements, use either coordinate derivative or normalderivatives on edges, and coordinate second derivative in the corner.Degrees of freedom on the edges and corners transform and map in adifferent way, different structure, solver more involved.



C1 continuous elements for Helmholtz equation

Left: one-dimensional example. Right: two-dimensional example.

0 5 10 15 20 25 30 35 40 45 50−14

−12

−10

−8

−6

−4

−2

0

2c1hh1a with alhh=1, nx=2, usym=exp(−x2) cos(7π/2 x) − 1d helmholtz

degree k

log

10(|

err

or|

∞,G

LL)

C0 elements

C1 elements

0 5 10 15 20 25 30 35 40 45 50−14

−12

−10

−8

−6

−4

−2c1hh2da with alhh=1, nx=2, usym=exp(−x2) cos(π/2 x) sin(π y) − 2d helmholtz

degree k

log

10(|

err

or|

∞,G

LL)

C0 elements

C1 elements



Mapped elements

• Straight-line quadrilaterals require bilinear element mappings, generalisoparametric elements represent mapping in spectral element basis.With these mappings, Jacobian and all needed derivatives of (inverse)mapping can be written as combinations of tensor-product matrices(corresponding to matrix-matrix multiplication) and diagonal matrices(corresponding to element-wise multiplication).

• Application of these matrices, and also of the stiffness and mass matrix,to a vector is therefore fast and runs close to peak, even thoughsomewhat more involved. Element-wise preconditioners are also more orless straightforward.

• Solution by static condensation (computing the Schur complement withrespect to the interface variables) and then solving the Schur complementsystem with some standard solver such as SuperLU.



Example 1 - 3 elements, approximation of Laplacian

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

5 10 15 20 25 30 35 40 45 50−14

−12

−10

−8

−6

−4

−2

0

N − degree of polynomial

log 10

(|re

sidu

al| ∞

)

Approximation for the weak laplacian, setupelem1, usym=sin(πx2)cos(πy)

interior residual for element 1interior residual for element 2interior residual for element 3residual for interface values



Example 2 - 12 elements

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2



Implementation in C

• Simple idea: code all the matrix operations and algorithms that arealready written in high-level linear algebra form in my MATLAB codenow in C using LAPACK, BLAS and other libraries as needed.

• Figuring out compilation and optimization for vendor specific librariesis interesting, and MATLAB does a lot of things (like eigenvaluecomputations and other matrix operations) very efficiently and withlot of hidden details that have to be coded explicitly when usingLAPACK/SUNPERFLIB/ATLAS etc.

• By now, there is a reasonable code base for SUN and for Linux Pentium,but I am still working on the optimization and on more complete runs.A skeleton version of the entire package runs, but visualization and moreextensive runs are not yet done.



Implementation in C - comparing errors (Sun Blade)

0 20 40 60 80 100 120 140 160 180 200−14

−12

−10

−8

−6

−4

−2gsylevprep/gsylvevsolv (MATLAB): Dirichlet problem for Laplace

log

10(m

ax−

err

or)

0 20 40 60 80 100 120 140 160 180 200−14

−12

−10

−8

−6

−4

−2gsylevprep/gsylvevsolv (C/sunperflib): Dirichlet problem for Laplace

log

10(m

ax−

err

or)



Implementation in C - results from my laptop

0

0.5

1

1.5

2

2.5

3

3.5

4

0 5 10 15 20 25 30 35 40 45 50

CP

U tim

e in s

econds

N

Time per solve

ATLAS+normal LAPACKATLAS+ATLASed LAPACK

linux/fedora rpm BLAS+LAPACK

0

1

2

3

4

5

6

0 5 10 15 20 25 30 35 40 45 50

CP

U tim

e in s

econds

N

Setup time

ATLAS+normal LAPACKATLAS+ATLASed LAPACK

linux/fedora rpm BLAS+LAPACK



Implementation in C - comparing timing (Sun Blade)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 5 10 15 20 25 30 35 40 45 50

CP

U tim

e in s

econds

N

Solve time on Sun Blade

Scaled time from laptopMATLAB on Sun Blade

Sun Blade, using ATLASSun Blade, using SUNPERF

0

10

20

30

40

50

60

70

80

90

0 5 10 15 20 25 30 35 40 45 50

CP

U tim

e in s

econds

N

Setup time on Sun Blade

Scaled time from laptopMATLAB on Sun Blade

Sun Blade, using ATLASSun Blade, using SUNPERF



Implementation in C - observations

• MATLAB is quite competitive against straigthforward coded C on mySun workstation, even using the vendor-optimized library SUNPERFLIB.SUNPERFLIB needs compiler and linker options to compile with theright library and to make it fast ...

• Properly optimized version using SUNPERFLIB and/or ATLAS shouldbe at least as fast as MATLAB (which uses ATLAS). Just coding aninterface to SUNPERFLIB and using its BLAS and LAPACK functionsdid not give a faster code in my case. Hunting down linker options anddirty details improved the speed of the C version to better than MATLAB

• C version on my laptop is quite fast. ATLAS seems to make a realdifference. Still playing around with some possibilities for optimization.

• Still need to profile on the Sun, test for other machines, and someoptimizations on all platforms.



Numerical observations

• For some degrees and some kinds of interpolation, see stronger gradientsin the vicinity of element interfaces. Stable, relatively local effect.Different kinds of filering are more or less successfull, a L2 projectioninto the C1 spectral element space seems to get rid of the edge effects,but makes the results more blurry.

• Poisson brackets: Higher order integration does seem to make a differenceand give results with sharper and finer strutures, but there is notmuch difference between projected continuous derivatives and piecewisecontinuous derivative.

• “Real” C1 elements seem to give quite encouraging preliminary resultsin the regular case.



Current and future work

• Possibly fully implicit time-stepping.

• Other problems: tearing mode, coalescence, Kelvin-Helmholtz, ...

• Move toward inclusion of some modules into M3D.

• Complete all the program versions: C1 version, mapped version, Cversion.

• Parallelization.


Spectral Elements for Two-Dimensional Resistive MHDhientzsc/losalamos.pdf · 2006-07-03 ·...

Documents

Transcript of Spectral Elements for Two-Dimensional Resistive MHDhientzsc/losalamos.pdf · 2006-07-03 ·...