Petascale Direct Numerical Simulations of Turbulent ... Overview Project Title Petascale Direct...
Transcript of Petascale Direct Numerical Simulations of Turbulent ... Overview Project Title Petascale Direct...
Petascale Direct Numerical Simulations ofTurbulent Channel Flow
MyoungKyu [email protected]
Department of Mechanical EngineeringUniversity of Texas at Austin
ESP Meeting May, 2013
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 1 / 30
Contents
Project Overview
Performance Optimization
Early Result
Conclusion
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 2 / 30
Project Overview
Project Title◮ Petascale Direct Numerical Simulations of Turbulent Channel Flow
Goal◮ Expanding our understand of wall-bounded turbulence
Personnel◮ P.I. : Robert Moser◮ Primary Developer : M.K.Lee◮ Software Engineering Support : Nicholas Malaya◮ Catalyst : Ramesh Balakrishnan
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 3 / 30
Turbulent Flow
We are living with turbulent flows.◮ Atmosphere◮ Transportation (Automobile, Airplane)◮ Blood flow in the heart
Directly related with energy loss by drag of fluid motion
Deterministic Chaos◮ Governing equation : Navier-Stokes equation◮ Analytic solution is unknown◮ Highly nonlinear / Unpredictable◮ Requires statistical approach
"The most important unsolved problem of classical physics" byRichard Feynman
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 4 / 30
Overlap Region
Connection between near-wall region and outer region
Not understood as well as near-wall region
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 5 / 30
History
1 10 100 1000y+
0
1
2
3
4
5
6
7y+
(dU
+/d
y+)
Re_tau = 180Re_tau = 590Re_tau = 950Re_tau = 2003
1 10 100 1000y+
0
5
10
15
20
25
30
U+
Re_tau = 180Re_tau = 590Re_tau = 950Re_tau = 2003
Reτ = 180 : Kim, Moin & Moser, 1987
Reτ = 590 : Moser, Kim & Mansour, 1999
Reτ = 940 : Del Álamo, Jiménez, Zandonade & Moser, 2004
Reτ = 2003 : Hoya & Jiménez, 2006
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 6 / 30
Computation of Overlap Region
High Reynolds(Re) number is required to obtain logarithmic layer.
Even with existing channel flow DNS data at the high Re,Reτ ≈ 2000, the region of log layer is small.
To expand our understanding of log layer, DNS data atReτ > 4000 ∼ 5000 is required.
Cost scales with Re4
Simulation at high Re is limited by computational power.
BG/Q enables us to simulate such a high Re flow
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 7 / 30
Simulation Geometry
Flow between two infinite parallel plates
Reτ ≈ 5000
Periodic boundary condition in streamwise / spanwise directions
No slip, no mass flux boundary conditions at wall
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 8 / 30
Simulation Geometry
Nx = 10,240 (Uniform) - DFT
Ny = 1,536 (Stretched, denser near wall) - B-SplineNz = 7,680 (Uniform) - DFT
242 Billion D.O.F.
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 9 / 30
Incompressible Navier-Stokes Equation
∂ui
∂t= −
∂P∂xi
+ Hi + ν∂2ui
∂xj∂xj,
∂uj
∂xj= 0
where
H1 = −
(
∂u2
∂x+
∂uv∂y
+∂uw∂z
)
H2 = −
(
∂uv∂x
+∂v2
∂y+
∂vw∂z
)
H3 = −
(
∂uw∂x
+∂vw∂y
+∂w2
∂z
)
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 10 / 30
Formulation (Kim, Moin & Moser, 1987)
Applying Laplacian and Curl operator
∂ωy
∂t= hg + ν∇2ωy
∂φ
∂t= hv + ν∇2φ
where
φ = ∇2v , At wall ωy = 0, v =
∂v∂y
= 0
hg =∂H1
∂z−
∂H3
∂x, hv =
∂2H2
∂x2 +∂2H2
∂z2 −∂2H1
∂x∂y−
∂2H3
∂y∂z
Two equations in the same form
No pressure term!!
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 11 / 30
Formulation (Spalart, Moser, & Rogers, 1991)
Time discretization : Low-Storage RK3 method
u′ = un +∆t[
L(
α1un + β1u′)
+ γ1Nn]
u” = u′ +∆t[
L(
α2u′ + β2u”)
+ γ2N ′ + ζ2Nn]
un+1 = u” + ∆t[
L (α3u” + β3un+1) + γ3N” + ζ3N ′]
Need only two additional fields
L : Linear operator (Laplacian) → Implicit + Explicit
N : Nonlinear Terms (hg ,hv ) → Explicit
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 12 / 30
Simulation Process
Periodic boundary condtion in X-dir and Z-dir◮ Fourier Transform◮ ∂/∂x → ikx , ∂/∂z → ikz◮ Ordinary differential equations in time
hg, hv : Quadratic nonlinear terms◮ Computed in realspace◮ Dealiasing : require 1/2 additional grid points
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 13 / 30
Simulation Process - Pencil Decomposition
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 14 / 30
For one time step
* 8 Racks, 1 MPI task/node
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 15 / 30
Solving Navier-Stokes Equation
B-Spline for y dir derivatives
Nx = 18432, Ny = 1536, Nz = 12288
131k 262k 393k 524k 786k
Number of Cores1
10
Ela
psed
tim
e (s
ec) Measured
Ideal100%
101%
102%
99.9%
96.5%
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 16 / 30
Discrete Fourier Transform
FFTW 3.3
Nx = 18432, Ny = 1536, Nz = 12288
131k 262k 393k 524k 786k
Number of Cores1
10
Ela
psed
tim
e (s
ec) Measured
Ideal100%
91.1%
93.6%
87.7%
89.6%
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 17 / 30
Data Transpose
FFTW 3.3 - MPI
Nx = 18432, Ny = 1536, Nz = 12288
131k 262k 393k 524k 786k
Number of Cores2
4
8
16
32
Ela
psed
tim
e (s
ec) Measured
Ideal100%
98.6%
100%98.6%
99.6%
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 18 / 30
Comparison with P3DFFT
parallel 3D FFT : Data transpose + FFT
ESP-FFTP3DFFT 2.5.1
◮ Most popular parallel 3D FFT library◮ Pencil decomposition
Differences (P3DFFT / ESP-FFT)◮ Communication Pattern : Fixed / Various◮ Odd-Ball wavenumber : Stored / Removed◮ (Buffer) Memory requirement : 3× / 1×◮ Threading capability : None / OpenMP (Data rearrangement, FFT)
* For benchmark, data is Fourier transformed in only x and z dir.
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 19 / 30
Comparison with P3DFFT (Small size)
Nx = 2048, Ny = 1024, Nz = 1024
128 256 512 1024 2048 4096 8192
Number of Cores0.01
0.1
1
10
100E
laps
ed ti
me
(sec
)P3DFFT (Measured)P3DFFT (Ideal)ESP-FFT (Measured)ESP-FFT (Ideal)
100%98.0%
97.7%98.6%
99.5%100%
101%
100%96.8%
115%116%
117%121%
123%
2.14 2.12 2.51 2.52 2.52 2.59 2.61
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 20 / 30
Comparison with P3DFFT (Large size)
Nx = 18432, Ny = 12288, Nz = 12288
65k 131k 262k 393k 524k 786k
Number of Cores1
10
Ela
psed
tim
e (s
ec)
P3DFFT (Measured)P3DFFT (Ideal)ESP-FFT (Measured)ESP-FFT (Ideal)
100%81.4%
89.6%
90.5%
100%
94.1%
89.5%
86.9%94.3%
81.5%
1.45 1.73 1.71 1.46
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 21 / 30
One full timestep
Nx = 18432, Ny = 1536, Nz = 12288
65k 131k 262k 393k 524k 786k
Number of Cores
10
100E
laps
ed ti
me
(sec
)MPI onlyHybrid (MPI + OpenMP)
1.19 1.13 1.02 1.14 1.00
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 22 / 30
Mean Velocity Profile
0 1000 2000 3000 4000 5000Y+
0
5
10
15
20
25
30
U+
1 10 100 1000Y+
0
5
10
15
20
25
30
U+
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 23 / 30
Variance & covariance of velocity components
0 1000 2000 3000 4000 5000Y+
-2
0
2
4
6
8
10
<uu
><uu><vv><ww><uv>
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 24 / 30
Energy Spectra
1 10 100 1000kx
1e-08
1e-06
0.0001
0.01
1
E_u
u
E_uuE_vvE_ww
1 10 100 1000kx
1e-12
1e-08
0.0001
1
E_u
u
E_uuE_vvE_ww
1 10 100 1000kz
1e-08
0.0001
1
E_u
u
E_uuE_vvE_ww
1 10 100 1000kz
1e-12
1e-08
0.0001
1
E_u
u
E_uuE_vvE_ww
* Left : Y+ = 10.5 / Right : Y+ = 4800
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 25 / 30
Energy Spectra (wavenumber multiplied)
0.1 1 10 100 1000kx
0
0.1
0.2
0.3
0.4
0.5
0.6
kE_u
u
kE_uukE_vvkE_ww
0.1 1 10 100 1000kx
0
0.025
0.05
kE_u
u
kE_uukE_vvkE_ww
0.1 1 10 100 1000kz
0
0.5
1
1.5
2
2.5
3
3.5
kE_u
u
kE_uukE_vvkE_ww
0.1 1 10 100 1000kz
0
0.05
0.1
0.15
0.2
0.25
kE_u
u
kE_uukE_vvkE_ww
* Left : Y+ = 10.5 / Right : Y+ = 4800
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 26 / 30
Streamwise Velocity
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 27 / 30
Spanwise Vorticity
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 28 / 30
Summary and future work
Summary◮ High Re DNS is required to expand our understanding of wall
bounded turbulent flow◮ Excellent scalability is achieved◮ Preliminary results are demonstrated
Future Work◮ Gathering more data until statistically converged◮ Investigating the multi-scale dynamics that governs the interaction
of the inner and outer turbulence◮ Upon completion of simulation, simulation data will be publicly
available at⋆ http://turbulence.ices.utexas.edu
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 29 / 30
THANK YOU!!
Questions?
M.K. Lee (Univ of Texas, Austin) Petascale DNS of Turbulent Channel Flow ESP Meeting May, 2013 30 / 30