Parallel Particle Simulation in Multiscale Fluid ... · Bart Verleye 8 July 2014,...
Transcript of Parallel Particle Simulation in Multiscale Fluid ... · Bart Verleye 8 July 2014,...
Technische Universitat Munchen
Parallel Particle Simulation inMultiscale Fluid Representations
SuperMUC Review Workshop
Philipp Neumann, Tobias Weinzierl, Denis Jarema,Bart Verleye
8 July 2014,[email protected]
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 1
Technische Universitat Munchen
Motivation
1 Multi-level fluid-particle treatment2 Many-particle systems:
Spacetree algorithms3 Outlook: Walking down the scales
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 2
Technische Universitat Munchen
Lattice Boltzmann–Navier-Stokes: ModelsNavier-Stokes Lattice Boltzmann
Mesh-based Mesh-based
Fluid in equilibrium Fluid close to equilibrium
DoF: velocities u, pressure p DoF: distribution functions fi
∇ · u = 0∂tu + u · ∇u = −∇p + 1
Re ∆u fi (x + cidt , t + dt) = fi (x, t) + ∆i (f − f eq)
Navier-Stokes
Lattice Boltzmann
ρ(x, t) =Q∑
i=1fi
p(x, t) = ρ(x, t)/c2s
u(x, t) = 1ρ(x,t)
Q∑i=1
fici
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 3
Technische Universitat Munchen
From Navier-Stokes to Lattice Boltzmann• Given: ρ, u, ∂xβuα + ∂xαuβ• Generally, it holds
Q > 1︸︷︷︸ + D︸︷︷︸ + D(D + 1)/2︸ ︷︷ ︸mass momentum stresses
• Split fi = f eqi (ρ,u) + f neq
i
• Ansatz1: continuum↔ non-equ. parts as small as possible
minf neq∈RQ
g(f neq) such that
∑i f neq
i = 0 mass∑i f neq
i ciα = 0 momentum∑i f neq
i ciαciβ = −c2s τ(∂xβuα + ∂xαuβ
)stresses
1 P. Neumann, H.-J. Bungartz, M. Mehl, T. Neckel, and T. Weinzierl. Commun. Comput. Phys. 12(1):65–84, 2012.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 4
Technische Universitat Munchen
From Navier-Stokes to Lattice Boltzmann• Given: ρ, u, ∂xβuα + ∂xαuβ• Generally, it holds
Q > 1︸︷︷︸ + D︸︷︷︸ + D(D + 1)/2︸ ︷︷ ︸mass momentum stresses
• Split fi = f eqi (ρ,u) + f neq
i
• Ansatz1: continuum↔ non-equ. parts as small as possible
minf neq∈RQ
g(f neq) such that
∑i f neq
i = 0 mass∑i f neq
i ciα = 0 momentum∑i f neq
i ciαciβ = −c2s τ(∂xβuα + ∂xαuβ
)stresses
1 P. Neumann, H.-J. Bungartz, M. Mehl, T. Neckel, and T. Weinzierl. Commun. Comput. Phys. 12(1):65–84, 2012.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 4
Technische Universitat Munchen
Application: Hierarchical Particle TransportCoarse-scale model: NS (fluid) + Faxen2 (particle)
NS Level 0
NS Level 1
NS Level 2LB Level 3
Fine-scale model: LB + Two-way coupledparticle solver3
2 H. Faxen. Appelberg, 1921.3 A.J.C. Ladd. J. Fluid Mech., 271:285–339, 1994.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 5
Technische Universitat Munchen
Particle Displacement in Channel Flow (1)
NS Level 0
NS Level 1
NS Level 2LB Level 3
• Start with LB, Level 3Switch to
• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80
• Compare to dynamically adaptive LB solution4
LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7
dtNS = 3.0e-5
4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 6
Technische Universitat Munchen
Particle Displacement in Channel Flow (2)
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.61.500
2.000
2.500
3.000
3.500
4.000
4.500Pos_X (LB-NS,0.75x0.35)
Pos_X (LB-NS,1.5x0.7)
Pos_X (LB-NS,3.0x1.4)
Pos_X (LB-NS,4.0x1.4)
Pos_X (LB)
posi
tion
(x-d
irect
ion)
time
• Start with LB, Level 3Switch to
• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80
• Compare to dynamically adaptive LB solution4
LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7
dtNS = 3.0e-5
4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 7
Technische Universitat Munchen
Particle Displacement in Channel Flow (2)
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.61.500
2.000
2.500
3.000
3.500
4.000
4.500Pos_X (LB-NS,0.75x0.35)
Pos_X (LB-NS,1.5x0.7)
Pos_X (LB-NS,3.0x1.4)
Pos_X (LB-NS,4.0x1.4)
Pos_X (LB)
posi
tion
(x-d
irect
ion)
time
• Start with LB, Level 3Switch to
• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80
• Compare to dynamically adaptive LB solution4
LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7
dtNS = 3.0e-5
4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 7
Technische Universitat Munchen
Particle Displacement in Channel Flow (2)
0.0064 0.0084 0.0104 0.0124 0.01441.500
1.502
1.504
1.506
1.508
1.510
1.512
1.514Pos_X (LB-NS,0.75x0.35)
Pos_X (LB-NS,1.5x0.7)
Pos_X (LB-NS,3.0x1.4)
Pos_X (LB-NS,4.0x1.4)
Pos_X (LB)
posi
tion
(x-d
irect
ion)
time
• Start with LB, Level 3Switch to
• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80
• Compare to dynamically adaptive LB solution4
LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7
dtNS = 3.0e-5
4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 7
Technische Universitat Munchen
Algorithmic Requirement Analysis
• (Transient) Mapping from particles to grid and vice versa• Linked lists/ cells: inappropriate→ particles may move more than one cell
• For few particles: global scatter/gather of particle position→ each rank holds all particle information
• For many particles: localised approach→ only domains which hold particle maintain particle data
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 8
Technische Universitat Munchen
Particle-in-Tree: Algorithm
• Make each cell of spacetree holdarray of particles (dynamic, most oftime empty)
• Make physics move particle,i.e. update particle positions (localcomputation within cell)
• Before cell is left: check whether(moved) particle still is within cell.Otherwise lift it to next coarser level
• Apply lift checks recursively• Before algorithm descends in spacetree: Check whether refined cell holds particles.
If so, sieve them into the children (drop)• Apply drops recursively
Remarks:• Particles belonging to level ` are never dropped deeper than level ` (drop constraint)• Benchmarking difficulties: Density, speed, mesh width...
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 9
Technische Universitat Munchen
Particle-in-Tree: Scaling
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 10
Technische Universitat Munchen
Outlook: Lattice Boltzmann–Molecular Dynamics
ParticleInsertion
MomentumInsertion
MacroscopicCellServiceMacroscopicCellService
MomentumController
KineticEnergyController
MoleculeWrapper
MoleculeIterator
MDSolverInterfaceMacroscopicSolverInterface
MDSimulation ContinuumSimulation
implements
utilises
utilises
sendsdata to
receivesdata from/
sendsdata to
implements
utilises utilises
MaMiCo
User/Programmer
uses/provides newimplementation of
TransferStrategy
Goal:
• Resolve regions of interest by molecular dynamics• Resolve other regions by Lattice Boltzmann• Partitioned approach: Macro-Micro-Coupling Tool5
→ Separation: molecular dynamics↔ Mesh-based solver
5 P. Neumann and N. Tchipev. Proceedings of ISPDC, 2012.
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 11
Technische Universitat Munchen
LB–MD: Channel FlowLB codes:• Peano framework6
• waLBerla7
• OpenLB (in preparation)
MD codes:• Simple MD• LAMMPS8
• ESPResSo9 (inpreparation)
• MarDyn (planned)
→ new SuperMUC project
Massively Parallel MultiscaleSimulation of Nanoflows
0 13 26 39 52 65 78 91 104 117 1300
0.2
0.4
0.6
0.8
1
Velocity (LBMD)Velocity (LB)
y
Vel
ocity
MD LBLB
• 3D steady-state coupling→ velocity relaxation
• LB: 54× 54× 54 cells, dx = 2.5
• MD: 105 LJ atoms
• MD: periodic boundaries+ buffer regions
6 www.peano-framework.org 7 www.walberla.net 8 lammps.sandia.gov 9 espressomd.org
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 12
Technische Universitat Munchen
Thank You...
... for your attention!
P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations
SuperMUC Review Workshop, 8 July 2014, [email protected] 13