Post on 07-Sep-2018
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Development of spectral element code for incompressible fluid
flows
PPééterter KKáávrvráánn
16th AER Symposium16th AER SymposiumBratislava, SlovakiaBratislava, Slovakia
2525--29 29 SpetemberSpetember 20062006
22
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
OverviewOverviewThe aim of the project,The aim of the project,Spectral element method,Spectral element method,NavierNavier--Stokes solver,Stokes solver,Implementation issues,Implementation issues,ResultsResults
33
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
The aim of the projectThe aim of the project
To develop an accurate To develop an accurate NavierNavier--Stokes solver, which isStokes solver, which is
based on spectral element method,based on spectral element method,
applicable to arbitrary 2D and 3D geometry,applicable to arbitrary 2D and 3D geometry,
arbitrary arbitrary DirichletDirichlet and Neumann boundary conditions can be and Neumann boundary conditions can be implemented into it ,implemented into it ,
applicable largeapplicable large--scale problemsscale problems
44
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
FiniteElementMethod
SpectralMethod
SpectralElementMethod
Spectral Element MethodSpectral Element MethodSpectral element method (SEM) is a subset of finite element methSpectral element method (SEM) is a subset of finite element methods, and ods, and
a a multidomainmultidomain spectral method,spectral method,
∑∑= =
=N
m
N
n
Nn
Nm
emn
eN llu),(u
1 1)()( ηξηξ
DifferencesDifferences between between SEMSEM and traditional and traditional FEMFEM::
Traditional Traditional FEMFEM uses lowuses low--order interpolation order interpolation functions,functions,
SEMSEM uses highuses high--order order LagrangianLagrangian interpolation interpolation functions based on functions based on JacobiJacobi--polinomialspolinomials::
basis Lobatto-Legendre-Gauss))(()1(
)()1()(2
mmN
NNm LNN
Llξξξ
ξξξ−+
′−−=
2D basis function, N=10
55
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
in higher dimensions bases are the tensorin higher dimensions bases are the tensor--product of the oneproduct of the one--dimensional basesdimensional bases
in 2Din 2D--> rectangular, in 3D > rectangular, in 3D --> > cuboidcuboid elements,elements,
computations are made on reference computations are made on reference elementselements
reference elements are mapped to the reference elements are mapped to the deformed elements using deformed elements using isoparametricisoparametricmapping.mapping.
good skew mesh tolerance, good skew mesh tolerance,
ξ
-1 0 1-1
0
1η
2D reference element2D reference element
GaussGauss--LegendreLegendre--LobattoLobatto pointspoints
66
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Some propertiesSome properties: :
Operator Costs (3D):Operator Costs (3D):
Standard FEMStandard FEM** SEMSEM# memory accesses: O( EN 6 ) O( EN 3 )# operations: O( EN 6 ) O( EN 4 )
N = order, E = number of elements, EN 3 = number of gridpoints
handles handles complex geometrycomplex geometry, ,
Spatial Spatial discretizationdiscretization error decays exponentiallyerror decays exponentially with the order of the with the order of the LagrangianLagrangianbases,bases,matrixmatrix--vector products: vector products:
⎯⎯ discretizeddiscretized operatorsoperators in spectral element method represented by in spectral element method represented by tensortensor--product matricesproduct matrices
⎯⎯ matrixmatrix--vector products cast as efficient matrixvector products cast as efficient matrix--matrix products:matrix products:
Ty
x
UDuIDuD
DUuDIuD
=⊗=
=⊗=
)(
)(Elemental derivative operator:
77
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Spectral element method for incompressible fluid flowsSpectral element method for incompressible fluid flows• TimeTime--dependentdependent NavierNavier--Stokes equation:Stokes equation:
•• use of Puse of PNN--PPNN--22 methodmethod:
P=5 element gridP=5 element grid
⎯⎯for theoretical reasons use staggered grid to avoid for theoretical reasons use staggered grid to avoid unphysical solutionsunphysical solutions
––velocity is approximated by P=velocity is approximated by P=NN order Gaussorder Gauss--LegendreLegendre--LobattoLobatto polinomialspolinomials,,
– pressurepressure is approximated by P=is approximated by P=NN--2 2 order Gaussorder Gauss--LegendreLegendre polinomialspolinomials.
⎪⎭
⎪⎬
⎫
=
++⋅−=+∂
∂
0
1)(
Du
MfDLuuuCu
M pTt ρ
ν≡=⋅∇
+∇−Δ=∇⋅+∂∂
⎪⎭
⎪⎬
⎫
0u
uuuu fpt ρ
ν 1
88
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Temporal Temporal discretizationdiscretization::
Temporal Temporal discretizationdiscretization is based on Operatoris based on Operator--IntegrationIntegration--Factor Splitting Factor Splitting Method (OIF),Method (OIF),
the main idea is to split operators and integrate them separatethe main idea is to split operators and integrate them separately,ly,
convection termconvection term expressed as material derivative, and expressed as material derivative, and discretizeddiscretized by 2nd by 2nd order backward differencing scheme:order backward differencing scheme:
)~(2
~3~4~ 12n
nnnS
tuuuu
=Δ
+− −− qn−u~ -- pure convection term computedpure convection term computed inin
by 4thby 4th--RungeRunge--KuttaKutta],[ nqn tt −
The The discretizeddiscretized unstable Stokes unstable Stokes --systemsystem to be solved implicitly:to be solved implicitly:
⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟
⎟⎠
⎞⎜⎜⎝
⎛
⎥⎥⎦
⎤
⎢⎢⎣
⎡
−−
0
nnT Bfu0DDH
np
)~( nS u -- Stokes operatorStokes operator
.23
0 ,0 =Δ+−= ⎟⎠⎞
⎜⎝⎛ β
βν MLH t -- HelmholtzHelmholtz--matrixmatrix
D , DT -- divergence, gradient operatordivergence, gradient operator
99
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
After applying the splitting method to the system After applying the splitting method to the system (according to(according to Paul Fischer: An Paul Fischer: An overlapping Schwarz method for spectral element solution of overlapping Schwarz method for spectral element solution of NavierNavier--Stokes equations, Stokes equations, J. of Comp. J. of Comp. Phys. Phys. 133,133, 8484--101), 101),
The algorithm to solve the StokesThe algorithm to solve the Stokes--problem (for a problem (for a dd--dimensional system):dimensional system):
nnt ppp −== +−Δ 1 ,0
δβT1DDME
1n1n MfDuH ++ += npT~
1nuDE +−= ~pδ
1nT11n uDMu +−Δ+ += ~0
pt δβ
ppp nn δ+=+1
1.1.
2.2.
3.3.
4.4.
dd HelmholtzHelmholtz--systemsystem
11 Pressure correction stepPressure correction step
The The d+1d+1 equations are solved by equations are solved by Preconditioned Conjugate Gradient Preconditioned Conjugate Gradient methodmethod
1010
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Implementation in C++ language,Implementation in C++ language,
objectobject--oriented designoriented design::
defining highdefining high--level data types, that represents geometrical, and algebraic level data types, that represents geometrical, and algebraic objects,objects,
operationsoperations are evaluated on are evaluated on abstract mathematical objectsabstract mathematical objects,,
the geometry and the linear algebra are decoupled,the geometry and the linear algebra are decoupled,
linear solverslinear solvers are are geometrygeometry (and problem) (and problem) independentindependent, ,
Example: Example: LinearSolver::PCG(MatrixSystem* mat_sys, Vector* lhs, Vector* rhs, Vector* BC);
use of standard libraries in mathematical modulesuse of standard libraries in mathematical modules::
-- BLAS (Basic Linear Algebra Subroutines)BLAS (Basic Linear Algebra Subroutines)
-- PETScPETSc (Portable Extensible Toolkit for Scientific Computing) (Portable Extensible Toolkit for Scientific Computing)
Code ImplementationCode Implementation
1111
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Geometry GenerationGeometry Generation
treat arbitrary 2D and 3D geometries, and treat arbitrary 2D and 3D geometries, and
approximate curved geometry as good as possibleapproximate curved geometry as good as possible
mesh generation must be usermesh generation must be user--friendly friendly
The aimThe aim::
Realization:Realization:
Coarse finite element mesh is generated by Coarse finite element mesh is generated by GAMBIT GAMBIT meshmesh--generation program,generation program,
the code can handle 4the code can handle 4-- and 9and 9--node quadrilateral, and 8node quadrilateral, and 8-- and 27and 27--node hexahedral node hexahedral elements elements
SEM SEM intrinsicallyintrinsically uses quadrilateral (2D) and hexahedral (3D) elementsuses quadrilateral (2D) and hexahedral (3D) elements
code reads GAMBIT files in Neutral File Format (.code reads GAMBIT files in Neutral File Format (.neuneu) )
spectral element mesh is built on the coarse finite element meshspectral element mesh is built on the coarse finite element mesh
1212
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Parallel ImplementationParallel Implementationparallel implementation is based on parallel implementation is based on Domain Decomposition Method,Domain Decomposition Method,
divide the computational domain into subdivide the computational domain into sub--domains, containing a branch of elements,domains, containing a branch of elements,
SubSub--domain solverdomain solver
Process #4
SubSub--domain solverdomain solver
Process #3
SubSub--domain solverdomain solver
Process #2
Communication on the Communication on the boundaries of subboundaries of sub--domains,domains,
CommunicationCommunication
SubSub--domain solverdomain solver
Process #1
Automatic mesh partitioning with Automatic mesh partitioning with MetisMetis library routines (n process library routines (n process --> n partition),> n partition),
parallel communication is realized by MPI and parallel communication is realized by MPI and PETScPETSc librarieslibraries
Single Program Multiple Data approach,Single Program Multiple Data approach,
1313
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Basic FlowchartBasic Flowchart
FE mesh generationFE mesh generationGAMBITGAMBIT mesh generation, using 4mesh generation, using 4--, 9, 9--node quadrilateral, 8node quadrilateral, 8--, ,
2727--node hexahedral elementsnode hexahedral elementsset boundary condition typesset boundary condition typesreads reads GAMBIT neutral (.GAMBIT neutral (.neuneu) files) files
FE mesh partitioningFE mesh partitioningMesh partitioning (Mesh partitioning (MetisMetis),),number of partitions = number of number of partitions = number of processesprocessesdistribute distribute submeshessubmeshes among processesamong processes
Build SE meshBuild SE mesheach process builds a spectral element mesh on its FE each process builds a spectral element mesh on its FE
submeshsubmesh,,create parallel communication patterns, create parallel communication patterns,
Parallel CalculationsParallel Calculationsdefine boundary conditions, define boundary conditions, build discrete operators,build discrete operators,perform parallel calculations, perform parallel calculations, save resultssave results
1414
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Test casesTest casesKovasznayKovasznay flowflow
Analytical solution of 2D steadyAnalytical solution of 2D steady--state state NavierNavier--Stokes equation,Stokes equation,incorporates nonlinear terms, good test for the full incorporates nonlinear terms, good test for the full NavierNavier--Stokes equationStokes equationCheck spectral convergence of the SEMCheck spectral convergence of the SEM
22
2
44
Re2
Re
2/)1(
2/2sin
2cos1
πλ
ππλ
π
λ
λ
λ
−−=
−=
=
−=
x
x
x
ep
yev
yeu
relδmax
N - polinomial order
Spectral element convergence of velocitySpectral element convergence of velocity
1515
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
2D lid2D lid--driven cavitydriven cavity
2D cavity flow, 10x10, N=10 order elements, vorticity contours (left), pressure contours (center), velocity contours (right) at t=12. Re=1000.
V=1V=1
1616
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Flow directionFlow direction
SubchannelSubchannel
Direct Numerical Simulation of flows in a Direct Numerical Simulation of flows in a subchannelsubchannel of a rod bundleof a rod bundleComputational problem definition:Computational problem definition:
diameter of the rod= 9.1mmdiameter of the rod= 9.1mmdistance between rods=12.2mmdistance between rods=12.2mmlength of the computational domain = 10mmlength of the computational domain = 10mmeach sides of the hexagon is coupled periodicallyeach sides of the hexagon is coupled periodicallynono--slip wall boundary condition on the surface of the slip wall boundary condition on the surface of the
rod rod
triangular array of fuel rods triangular array of fuel rods
1717
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Coarse GAMBIT mesh of the Coarse GAMBIT mesh of the subchannelsubchannel
GAMBIT mesh: 27GAMBIT mesh: 27--node hexahedral node hexahedral elements in order to follow curved elements in order to follow curved geometry of the rodgeometry of the rod
10 layers, 72 elements in each layer 10 layers, 72 elements in each layer = 720 element= 720 element
SE mesh: expansion order N=12,SE mesh: expansion order N=12,
~ 1.2 * 10~ 1.2 * 1066 velocity nodes, ~ 9.5 * velocity nodes, ~ 9.5 * 10105 5 pressure nodespressure nodes
elements are concentrated at the elements are concentrated at the wall in order to resolve boundary layerwall in order to resolve boundary layer
1818
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Laminar flow around in a Laminar flow around in a subchannelsubchannel
Computed solution Computed solution Relative difference between approximate Relative difference between approximate analytical and computed results analytical and computed results
m/sm/s
relative difference is independent of expansion orderrelative difference is independent of expansion order
There is an approximate analytical solution for stationary lamiThere is an approximate analytical solution for stationary laminar flow in nar flow in triangular array of rodstriangular array of rods
1919
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Towards turbulent flow modelingTowards turbulent flow modelingDirect numerical simulation (DNS) solves Direct numerical simulation (DNS) solves NavierNavier--Stokes equations as accurate as possible Stokes equations as accurate as possible
without turbulence models,without turbulence models,
There are attempts for DNS simulation of turbulent flows in roThere are attempts for DNS simulation of turbulent flows in rod bundles:d bundles:
Paul Fischer uses SEM for DNS of Paul Fischer uses SEM for DNS of ReRe=30.000 flow in rectangular array,=30.000 flow in rectangular array,
Mayer Mayer GusztGusztáávv uses lattice uses lattice BoltzmannBoltzmann method for DNS and LES in triangular array,method for DNS and LES in triangular array,
DNS of turbulent flows in rod bundle helps us to understand mixDNS of turbulent flows in rod bundle helps us to understand mixing processes:ing processes:
Projection of mean lateral velocity to a cross-section in LES simulation using lattice Boltzmannmethod.
2020
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Numerical approachNumerical approach
common practice in periodic channel flow simulation is to use ccommon practice in periodic channel flow simulation is to use constantonstant--body body force term to accelerate the flow in flow direction,force term to accelerate the flow in flow direction,
our approach is to adjust forcing term to maintain constant fluour approach is to adjust forcing term to maintain constant flux in the axial x in the axial direction,direction,
tests are run with different expansion order and different meshetests are run with different expansion order and different meshes,s,
no statistical quantities are available yet,no statistical quantities are available yet,
flow of flow of ReRe=5000=5000
2121
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
m/s
Contours of velocity component in flow direction in a vertical cross-section.
Flow direction
m/s
Snapshot of the velocity field in the subchannel
2222
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Contours of velocity component in Contours of velocity component in flow direction, in a cross sectionflow direction, in a cross section
Lateral velocity vectors in a cross Lateral velocity vectors in a cross sectionsection
m/s
2323
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
ConclusionsConclusions
A simulation code for incompressible fluid flows has been develA simulation code for incompressible fluid flows has been developed,oped,
based on spectral element method,based on spectral element method,
capable of handling arbitrary complex geometries,capable of handling arbitrary complex geometries,
can be run on parallel, distributed memory architecture (computcan be run on parallel, distributed memory architecture (computerer--cluster),cluster),
implemented in c++, with object oriented design (~50000 lines oimplemented in c++, with object oriented design (~50000 lines of code)f code)
several test cases have been run,several test cases have been run,
tests of turbulent flows are runningtests of turbulent flows are running
Future work:Future work:
compare simulation results with available experimental data forcompare simulation results with available experimental data for flows Re~10000,flows Re~10000,
increase parallel efficiency, increase parallel efficiency,
2424
Hungarian Academy of Sciences Hungarian Academy of Sciences KFKI Atomic Energy Research InstituteKFKI Atomic Energy Research Institute
Thank you Thank you for your attentionfor your attention!!