Applications of Lattice Boltzmann Methods...2017/01/12 · The Lattice Boltzmann Method (LBM) 8 f q...

Applications of Lattice Boltzmann Methods

Dominik Bartuschat, Ulrich Rüde

New Delhi, IndiaJanuary 12, 2017

UGC-DAADLecture Series 2017

D. Bartuschat, M. Bauer, S. Bogner, C. Godenschwager, F. Schornbaum, U. Rüde

Chair for System Simulation, FAU Erlangen-Nürnberg

New Delhi, 12.01.2017 - Dominik Bartuschat - System Simulation Group - Applications of Lattice Boltzmann Methods(Contact: [email protected])

Outline

● The waLBerla Simulation Framework● The Lattice Boltzmann Method and Complex Flows● Fluid-Particle Interactions● Charged Particles in Fluid Flow● Free Surface Flow

3

mailto:[email protected]:[email protected]

The waLBerla Simulation Framework


waLBerla

● Widely applicable Lattice Boltzmann framework● Suited for various flow applications● Large-scale, MPI-based parallelization● Dynamic application switches for heterogeneous architectures and

optimization

5



waLBerla Concepts

6

Block concept:● Domain partitioned into Cartesian grid of blocks● Blocks can be assigned to different processes● Blocks contain:

● cell data, e.g. fluid density, electric potential● global information e.g. location, MPI rank

Communication concept:● Simple communication mechanism on uniform grids, utilizing MPI● Ghost layers to exchange cell data with neighboring blocks

Sweep concept:● Sweeps are work steps of a time-loop, performed on block-parallel level● Example: MG sweep, contains sub-sweeps (restriction, prolongation, smoothing)


The Lattice Boltzmann Method

and flow in complex geometries

● Discrete lattice Boltzmann equation ● Describes probabilities fq that fluid molecules move with given velocities● Molecule collisions represented by collision operator Wq

fq(~xi +~cqdt, tn+dt)� fq(~xi , tn) = Wq (fq(~xi , tn))


The Lattice Boltzmann Method (LBM)

8

fq

● Domain discretized into cubic cells● Discrete velocities and associated

distribution functions per cell~cq

D3Q19 modelIllustration by Klaus Iglberger


˜

fq(~xi , tn) = fq(~xi , tn)�1

t�fq(~xi , tn)� f eqq (~xi , tn)

�

| {z }⌦q with single relaxation time

fq(~xi +~eq, tn+dt) = f̃q(~xi , tn)


Stream-Collide

● Stream step:

9

The equation is solved in two steps:

● Collide step:

Fluid viscosity determined by , fluid velocity and density computable from nf t fq~u rf



●Sparse, but coherent geometry●Volume fraction 0.3%● Large number of small blocks●Multiple blocks per process● Load balancing required

Flow in Complex Geometries

10

Flow through Coronary Arteries



Complex Geometry Initialization

11



Domain Partitioning

● Domain partitioning of coronary tree dataset● Partitioning for aim of one block per process

● Excellent scaling on JUQUEEN with up to 1 trillion lattice cells*

12

JUQUEEN nodeboard512 processes, 485 blocks

JUQUEEN, full machine458 752 processes, 458 184 blocks

* C. Godenschwager et al. „A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries“ (2013), doi:10.1145/2503210.2503273

mailto:[email protected]:[email protected]://dx.doi.org/10.1145/2503210.2503273http://dx.doi.org/10.1145/2503210.2503273


Blood Flow and Perfusion● Flow simulation in arteries +

Myocardium as porous medium, modeled by LBM forcing term*● At aorta and endocardium: pressure boundary conditions● Blood flow visualization by particle tracing:

13

* Z. Guo, T. Zhao. „ Lattice Boltzmann model for incompressible flows through porous media“ (2002), doi:10.1103/PhysRevE.66.036304

mailto:[email protected]:[email protected]://dx.doi.org/10.1103/PhysRevE.66.036304http://dx.doi.org/10.1103/PhysRevE.66.036304


Perfusion Results

14


Fluid-Particle Interaction with LBM

and tumbling spherocylinders in Stokes flow


Fluid-Particle Interaction – 4-Way Coupling

● Particles mapped onto fixed lattice Boltzmann grid● Cells overlapped by object treated as moving boundary● Hydrodynamic forces on particle computed by momentum exchange method*

16

* N. Nguyen, A. Ladd. „Lubrication corrections for lattice-Boltzmann simulations of particle suspensions“ Phys. Rev. E (2002), doi:10.1103/PhysRevE.66.046708

Illustration by Jan Götz

mailto:[email protected]:[email protected]://dx.doi.org/10.1103/PhysRevE.66.046708http://dx.doi.org/10.1103/PhysRevE.66.046708

1/� =length

radius= 12


Tumbling Spherocylinders in Stokes Flow

● Tumbling motion of elongated particles in Stokes flow● Four spherocylinders in periodic domain, aspect ratio

● LBM simulations with TRT operator and comparison against slender body formulation (examining influence of inertia, wall effects, and particle shape)*

17

* D. Bartuschat et al. „Two Computational Models for Simulating the Tumbling Motion of Elongated Particles in Fluids“ (2016), doi:10.1016/j.compfluid.2015.12.010

mailto:[email protected]:[email protected]://dx.doi.org/10.1016/j.compfluid.2015.12.010http://dx.doi.org/10.1016/j.compfluid.2015.12.010


Two Tumbling Spherocylinders

● Flow Field around two spherocylinders,1/ε =12

18

50

100

150

200

250 ·10≠6

x[m

]

Á = 1/10 Á = 1/12 Á = 1/14

≠200

≠100

0

100

200·10≠6

u

x

[m/s

]

0 5 10 15 20 25

1.2

1.4

1.6

1.8

2

2.2·10≠3

t [s]

u

z

[m/s

]

● Motion dependent on aspect ratio 1/ε

➡ Convergence to preferred distance xmax due to inertia➡ Max. velocity uz for horizontal particle orientation,

min. uz for vertical orientation

Domain size: [576 dx]3

Fluid: Water (20°C)Time steps: 600000

dx=4.98µm, dt=4.55⋅10-5s, τ =6Particle density: 1492 kg/m3, radius = 4dxRuntime on LiMa: 16h, 768 cores


Charged Particles in Fluid Flow

for particle-laden electrokinetic flows


Motivation

20

Interactions of large numbers of charged particles in fluid flows, influenced by external electric fields

© Kang and Li „Electrokinetic motion of particles and cells in microchannels“ Microfluidics and Nanofluidics

● Industrial applications:● Filtering particulates from exhaust gases● Charged particle deposition in cooling

systems of fuel cells

● Medical applications:● Optimization of Lab-on-a-Chip systems:● Sorting of different cells● Trapping cells and viruses

● Deposition of charged aerosol particles in respiratory tract



Multi-Physics Simulation

21

Electro (quasi) statics

Fluiddynamics

Rigid body dynamics

hydrodynamic force

object movement

ion convectionforce on ions

electr

ostat

ic for

ce

charg

e den

sity



Poisson Equation and Force on Particles

22

��F(~x) = re (~x)ee

~FC =�qparticle ·—F(~x)

● Electric potential described by Poisson equationwith particle‘s charge density on RHS:

● Discretized by finite volumes on lattice● Solved with cell-centered multigrid solver

implemented in waLBerla● Supersampling for computing overlap degree

to set RHS more accurately

● Electrostatic force on particle:

re

Illustration by Jan Götz



6-Way Coupling for Charged Particles

23

hydrodynam. force

object motion

Lubricationcorrection

electrostat. force

velocity BCs

object distance

LBM

correction force

charge distribution

Newtonian mechanicscollision response

treat BCsstream-collide step

Finite volumes

MGiterat.

treat BCsV-cycle

D. Bartuschat, U. Rüde. „Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows“, J. Comput. Sci. (2015), doi:10.1016/j.jocs.2015.02.006

mailto:[email protected]:[email protected]://dx.doi.org/10.1016/j.jocs.2015.02.006http://dx.doi.org/10.1016/j.jocs.2015.02.006


Charged Particles Algorithm

24

foreach time step, do

// solve Poisson problem with particle charges:set RHS of Poisson equationwhile residual too high do

perform multigrid V-cycle to solve Poisson’s equation

// solve lattice Boltzmann equation// considering particle velocities:begin

set velocity BCs of particlesperform stream-collide step

// couple potential solver and LBM to pe:begin

apply hydrodynamic force to particlesapply electrostatic force to particlesperform lubrication correctionpe moves particles depending on forces


Charged Particles – Multigrid Solver

● Based on● Smoothing principle:

High-frequency error elimination by iterative solvers (e.g. GS)

● Coarse grid principle: Restriction to coarser grid transforms low-frequency error components to relative higher-frequency ones

● Smoothing on coarse grids● Prolongation of obtained correction

terms to finer grid

● Iterative method for efficient solution of sparse linear systems


26

● Applied recursively, V(νpre, νpost)-cycle

Multigrid


● All operations implemented as compact stencil operations

● Design goals:● Efficient and robust black-box solver● Handling complex boundary conditions on coarse levels● Naturally extensible to jumping coefficients

➡ Method of choice: Galerkin coarsening

● (FV) Stencils stored for each unknown● On finest level: quasi-constant stencils

● Averaging restriction, constant prolongation● Preserves D3Q7 stencil on coarse grids● Convergence rate deteriorates● Workaround for Poisson problem: Overrelaxing prolongation*


27

* Mohr, Wienands „Cell-centered Multigrid Revisited“, Comput. Vis. Sci. (2004)

P1 P2

P3 P4

Cell-Centered Multigrid - Implementation


Charged Particles – Validation

Radius: R = 600µmCharge: qe = 8000eSupersampling: factor 3

F(~r) = 14pee

qe|~r | if |

~r |� R


29

Validation of Electric Potential

Analytical solution for homogeneously charged particle in infinite domain:

0

50

100

150

200

250

0

0.1

0.2

0.3

·10�3

xL

F/V

Analytical solution

Numerical solution

Sphere surface

~rqe

R

x

Domain: [256 dx]3● Dirichlet BCs: exact solution● MG: 5 V(2,2)-cycles● Residual threshold 2⋅10-9



30

Determination of residual threshold:

Error hardly reduced after residual norm smaller than 10-9➡Residual threshold for simulations: 2⋅10-9

V(2,2) cyclesConv. rate 0.18

Validation of Electric Potential

0 1 2 3 4

5

10

�10

10

�9

10

�8

10

�7

10

�6

10

�5

Iteration

L 2n

orm

Error

Residual


Charged Particles – Results


32



Channel: 2.56mm⇥7.68mm⇥1.92mm Particles: R = 80µm, qe =±40 000e Charged plates: F =±76.8Vdx = 10µm,t = 1.7,dt = 40µs Water (20 �C), Inflow 1 mm/s, Outflow 0 Pa else: no-slip & insulating BCs



33

•Computed on 144 cores (12 nodes) of RRZE - LiMa

•210 000 time steps

•15.7h runtime•643 unknowns per core

•6 MG levels



34

Weak scaling:● Costant size per core:

● 1283 cells● 9.4% moving obstacle cells

● Size doubled (in all dimensions)● Cell-centered MG

● V(3,3) with 7 levels● 6 to 116 CG coarse-grid iterations● Convergence rate: 0.07

● 2x4x2 cores per node

Experiments on SuperMUC:● 18 thin islands with 512 compute nodes, each:

● 16 cores (2 Xeon chips) @2.5 GHz● 32 GB DDR3 RAM

● Ranked #6 in TOP500 during experiments

Parallel Scaling Experiments



35

Weak Scaling for 240 Time Steps

32 768 cores7.1M particles

1 2 4 8 16 32 64 128

256

512

1024

2048

0

100

200

300

400

Number of nodes

Tota

lrun

times/s

LBMMapLubrHydrFpeMGSetRHSPtCmElectF

Overall parallel efficiency @2048 nodes: 83%

D. Bartuschat, U. Rüde. „Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows“ (2015), doi:10.1016/j.jocs.2015.02.006



36

LBM 91 %MG - 1 V(3,3) 64 %

● Parallel efficiency @2048 nodes:

Weak Scaling for 240 Time Steps

➡ MG performance restricted by coarsest-grid solving

Meg

a la

ttice

site

upd

ates

per

sec

.

Meg

a flu

id la

ttice

site

upd

ates

per

sec

.

0 250 500 750 1000 1250 1500 1750 2000Number of nodes

0102030405060708090

103

MFL

UPS

(LB

M)

LBM Perform.20

40

60

80

100

120

103

MLU

PS (M

G)

MG Perform.

32 768 cores7.1M particles

D. Bartuschat, U. Rüde. „Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows“ (2015), doi:10.1016/j.jocs.2015.02.006


Electrophoresis of Charged Particles


Multi-Physics Simulation

38

Electro (quasi) statics

Fluiddynamics

Rigid body dynamics

hydrodynamic force

object movement

ion convectionforce on ions

electr

ostat

ic for

ce

charg

e den

sity



Electrical Double Layer (EDL)

● Surfaces in contact with most liquids acquire surface charge● Surface charge balanced by oppositely charged ions from fluid

39

��y = eee Âi

zini•e� zi eykBT zi : valence of ions, e: elementary charge,

ni•: bulk ionic number concentration

● EDL potential described by Poisson-Boltzmann equation

x

yy

s

yd z

lD

solid

Shear plane

Stern plane

+

+

+

+

+

+

+

+

+

�

�

��

��

�

��

+

��

+�

�

�

+

+

+

�

�

�

�

diffuse layer

x

yy

s

yd z

lD

lD

Characteristic parameters: - potential EDL thickness z


Lattice Boltzmann equation with body force term

fq(~xi +~cqdt, t+dt)� fq(~xi , t) = Wq +dtFq(~fb)


Electrostatic Force on Fluid in EDL

40

Electrostatic force on double layer charge~fb

= re

(y) ·~Eext

Electrical double layer potentialSymmetric Poisson-Boltzmann equation (PBE)

��y =�2z en•ee

sinh✓z eykB T

◆

Debye-Hückel approximation (DHA) for

��y =�k2 y, k = l�1D

|z |< 25mV


r

qy = z

R

lD

x

Radius: R = 120nmCharge: qs = −124eFluid: Water (20 �C)k R ⇡ 0.89


41

Domain: [128 dx]2 x 256 dx● SOR solver● Dirichlet BCs: exact solution● Residual reduction 2⋅10-7

Validation of Double Layer Potential

Analytical solution for sphere with uniform surface charge in infinite domain (Debye-Hückel approximation):

0 20 40

60

80 100 120

�10

�8

�6

�4

�2

0

·10�3

xL

y/V

Analytical

solution

Numerical

solution

y(~r) = z R|~r| e�k(|~r|�R) if |~r|� R


Channel: 1.28µm⇥2.56µm⇥1.28µm Particle: R = 120nm, qe = −124e z = −10mV, Ey =�4.7 · 10 6 V/mFluid: Water (20

�C), c• = 5µmol/l dx = 10nm,t = 6.5,dt = 0.2ns BCs: Periodic in axial direction,

else insulating & no-slip BCs


Electrophoresis in Micro-Channel

Electrophoresis of charged particle along channel axis● Result after 30 000 time steps● Flow field, double layer potential, and ion charge distribution

42

U⇤EP = 0.21ms


Free Surface Flow

and effects of surface tension


Free Surface Extension

44

●Volume of Fluid Approach*●Only liquid phase has to

be simulated**●Cells are classified as

either solid, liquid, gas or interface●LBM only performed in

fluid cells●Suitable for phases with

high density differences

*C. Hirt, B. Nichols. „Volume of Fluid (VOF) Method for the Dynamics of Free Boundaries“ (1981), doi:10.1016/0021-9991(81)90145-5**C. Körner et al. „Lattice Boltzmann Model for Free Surface Flow for Modeling Foaming“ (2005), doi:10.1007/s10955-005-8879-8

mailto:[email protected]:[email protected]://dx.doi.org/10.1016/0021-9991(81)90145-5http://dx.doi.org/10.1016/0021-9991(81)90145-5http://dx.doi.org/10.1007/s10955-005-8879-8http://dx.doi.org/10.1007/s10955-005-8879-8


Free Surface Extension

45

●Geometry Reconstruction:● computation of interface normals● compute normals at triple points● compute local curvature to account for

surface tension

●Surface Dynamics Simulation:● non-free-surface boundary treatment● free surface boundary treatment● LBM streaming step ( advection )● fill level updates ( mass advection )● LBM collision step● conversion of cell types



Drop on Inclined Plane

46

kin. viscosity:surf. tension:drop diameter:drop resolution:τ:dt:

7.9‧10-6 m2/s 4.82‧10-2 N/m5 mm100 cells 0.5262.8‧10-6 s

SRT with Guo forcing term*

* Z. Guo et al. „ Discrete lattice effects on the forcing term in the lattice Boltzmann method“ (2002), doi: 10.1103/PhysRevE.65. 046308

mailto:[email protected]:[email protected]://dx.doi.org/10.1103/PhysRevE.65.%20046308http://dx.doi.org/10.1103/PhysRevE.65.%20046308


Domain Setup

●Full free surface calculations are costly:● find interface cells● reconstruct interface normals and curvature● mass advection● cell conversions

47

●Restriction to L-shaped domain●No inclined geometry, use inclined gravity instead:

gravity



Domain Setup

●Full free surface calculations are costly●Only fraction of complete domain covered with fluid/interface●Fluid covered region is moving➠ Dynamic load balancing required:

48



Domain Setup

●Full free surface calculations are costly●Only fraction of complete domain covered with fluid/interface●Fluid covered region is moving➠ Dynamic load balancing required

49

●Simulation on 200 cores (on Emmy, RRZE)●Drop resolution of 100 cells➠ Runtime between 1 and 2 hours



Boundary Setup

●No-slip boundary: solid wall, enforces zero velocity at boundary●Pressure boundary: pressure Dirichlet (set to capillary pressure)

50



Varying Inclination Angle

51

15° 30° 45°

➠ Higher inclination causes drop to move further before being absorped



Varying Contact Angle

52

50° 90° 110°

➠ Higher contact angle causes drop to move further before being absorped


Free Surface Flow with Particles


Floating Objects

54

S. Bogner, U. Rüde. „ Simulation of floating bodies with lattice Boltzmann“ (2012), doi:10.1016/j.camwa.2012.09.012

mailto:[email protected]:[email protected]://dx.doi.org/10.1016/j.camwa.2012.09.012http://dx.doi.org/10.1016/j.camwa.2012.09.012


Rising Bubble

55

S. Bogner, U. Rüde. „ Simulation of floating bodies with lattice Boltzmann“ (2012), doi:10.1016/j.camwa.2012.09.012

mailto:[email protected]:[email protected]://dx.doi.org/10.1016/j.camwa.2012.09.012http://dx.doi.org/10.1016/j.camwa.2012.09.012

Thank you for your attention!

Applications of Lattice Boltzmann Methods...2017/01/12 · The Lattice Boltzmann Method (LBM) 8 f q...

Documents

Transcript of Applications of Lattice Boltzmann Methods...2017/01/12 · The Lattice Boltzmann Method (LBM) 8 f q...