Nonlinear Computational Aeroelasticity Lab

ACCELERATION OF A

COMPUTATIONAL FLUID DYNAMICS

CODE WITH GPU USING OPENACC

N I C H O L S O N K . KO U K PA I Z A NP H D . C A N D I D AT E

GPU Technology Conference 2018, Silicon ValleyMarch 26-29 2018

• GT NCAEL Team members

• N. Adam Bern

• Kevin E. Jacobson

• Nicholson K. Koukpaizan

• Isaac C. Wilbur

CONTRIBUTORS TO THIS WORK

• Mentors

• Matt Otten (Cornell University)

• Dave Norton (PGI)

• Advisor

• Prof. Marilyn J. Smith

2

• Initial work done at the Oak Ridge GPU Hackathon (October 9th-13th 2017)

• “5-day hands-on workshop, with the goal that the teams leave with applications running on GPUs, or at least with a clear roadmap of how to get there.” (olcf.ornl.gov)

HARDWARE

• Access to summit-dev during the Hackathon

• IBM Power8 CPU

• NVIDIA Tesla P100 GPU - 16 GB

• Access to NVIDIA’s psg cluster

• Intel Haswell CPU

• NVIDIA Tesla P100 GPU- 16 GB

3

Source: NVIDIA (http://www.nvidia.com/object/tesla-p100.html)

• Validated Computational Fluid Dynamics (CFD) solver

– Finite volume discretization

– Structured grids

– Implicit solver

• Written in Free format Fortran 90

• MPI parallelism

• Approximately 50,000 lines of code

• No external libraries

• Shallow data structures to store the grid and solution

APPLICATION: GTSIM

4Reference for GTSIM: Hodara, J. PhD thesis “Hybrid RANS-LES Closure for Separated Flows in the Transitional Regime.” smartech.gatech.edu/handle/1853/54995

• Explicit CFD solvers:

• Conditionally stable

• Implicit CFD solvers:

• Unconditionally stable

• Courant-Friedrichs-Levy (CFL) number dictates convergence and stability

WHY AN IMPLICIT SOLVER?

Source: Posey, S. (2015), Overview of GPU Suitability and Progress of CFD Applications, NASA Ames Applied Modeling & Simulation (AMS) Seminar – 21 Apr 2015 5

Read in the simulation parameters, the grid and initialize the solution arrays

Loop physical time iterations

Loop pseudo-time sub-iterations

Compute the pseudo-time step based on the CFL condition

Build the left hand side (𝑳𝑯𝑺) 40 %

Compute the right hand side (𝑹𝑯𝑺) 31%

Use an iterative linear solver to solve for Δ𝑼 in 𝑳𝑯𝑺 × Δ𝑼 = 𝑹𝑯𝑺24%

Check the convergence

end loop

end loop

Export the solution (𝑼)

PSEUDOCODE

6

• Write 𝑳𝑯𝑺 = ഥ𝓛 +𝓓 + ഥ𝓤

• Jacobi based (Slower convergence, but more suitable for GPU)

Δ𝑼𝑘 = ഥ𝓓−1 𝑹𝑯𝑺𝑘−1 − ഥ𝓛Δ𝑼𝑘−1 − ഥ𝓤 Δ𝑼𝑘−1

OVERFLOW solver (NAS Technical Report NAS-09-003, November 2009) used Jacobi for GPUs

• Gauss-Seidel based (one of the two following formulations)

Δ𝑼𝑘 = ഥ𝓓−1 𝑹𝑯𝑺𝑘 − ഥ𝓛 Δ𝑼𝒌 − ഥ𝓤 Δ𝑼𝒌−𝟏

Δ𝑼𝑘 = ഥ𝓓−1 𝑹𝑯𝑺𝑘 − ഥ𝓛 Δ𝑼𝒌−𝟏 − ഥ𝓤 Δ𝑼𝒌

• Coloring scheme (red - black)

• Red: Use the first Gauss-Seidel formulation, with previous iteration black cells data

• Black: Use the second Gauss-Seidel formulation with the last Red update

LINEAR SOLVERS (1 OF 3)

7

LINEAR SOLVERS (2 OF 3)

• LU-SSOR (Lower-Upper Symmetric Successive Overrelaxation) scheme • Coloring scheme (red-black)

8Coloring scheme is more suitable for GPU acceleration

Source: Blazek, J., Computational Fluid Dynamics: Principles and Applications. Elsevier, 2001.

Source: https://people.eecs.berkeley.edu/~demmel/cs267-1995/lecture24/lecture24.html

• What to consider with the red-black solver

• Coloring scheme converges slower than LU-SSOR scheme

• Need more linear solver iterations at each step

• Because of the 4th order dissipation, black also depends on black!

potentially even slower convergence

• Reinitializing Δ𝑼 to zero proved to be best

LINEAR SOLVERS (3 OF 3)

9

Is using a GPU worth the loss of convergence in the solver?

TEST PROBLEMS

• Laminar Flat plate

• 𝑅𝑒𝐿 = 10000

• 𝑀∞ = 0.1

• (2D): 161 x 2 x 65 Initial profile

• (3D): 161 x 31 x 65 Hackathon

• Other coarser/finer meshes to understand the scaling

• Define two types of speedup

• Speedup: comparison to a CPU for the same algorithm

• “Effective” speedup: comparison to more efficient CPU algorithm

10

• Port the entire application to GPU for laminar flows

• Obtain at least a 1.5 x acceleration on a single GPU compared to a

CPU node, (approximately 16 cores) using OpenACC

• Extend the capability of the application using both MPI and GPU

acceleration

HACKATHON OBJECTIVES AND STRATEGY (1 OF 2)

11

• Data

• !$acc data copy ()

• Initially, data structure around all ported kernels slowdown

• Ultimately, only one memcopy (before entering the time loop)

• Parallel loops with collapse statement

• !$acc parallel loop collapse(4) gang vector

• !$acc parallel loop collapse(4) gang vector reduction

• !$acc routine seq

• Temporary and private variables to avoid race conditions

• Example 𝑟ℎ𝑠 𝑖, 𝑗, 𝑘 , 𝑟ℎ𝑠(𝑖 + 1, 𝑗, 𝑘) updated in the same step

HACKATHON OBJECTIVES AND STRATEGY (2 OF 2)

12

• Speedup

• 13.7x versus single core

• 3.7x versus 16 core, but this MPI test did not exhibit linear scaling

• Initial objectives not fully achieved, but encouraging results

• Postpone MPI implementation until better speedup is obtained with the serial implementation

RESULTS AT THE END OF THE HACKATHON

GPU CPU (16 cores) - MPI CPU 1 core

6.5 sec 23.9 s 89.7 s

• Total run times (10 steps on a 161 x 31 x 65 grid)

13

• Now that the code runs on GPU, what’s next?

• Can we do better?

• What’s the cost of using the coloring scheme versus the LU-SSOR scheme?

• Improve loop arrangements and data management

• Make sure all !$acc data copy () statements have been replaced by !$acc data present () statements

• Make sure there are no implicit data movements

FURTHER IMPROVEMENTS (1 OF 2)

14

• Further study and possibly improve the speedup

• Evaluate the “effective” speedup

• Run a proper profile of the application running on GPU with pgprof

pgprof --export-profile timeline.prof ./GTsim > GTsim.log

pgprof --metrics achieved_occupancy,expected_ipc -o metrics.prof ./GTsim > GTsim.log

FURTHER IMPROVEMENTS (2 OF 2)

15

DATA MOVEMENT

• !$acc data copy() !$acc enter data copyin()/copyout()

• Solver blocks (𝑳𝑯𝑺, 𝑹𝑯𝑺) are not actually need back on the CPU

• Only the solution vector needs to be copied out

16

LOOP ARRANGEMENTS

• All loop in the order k, j, I

• Limit the size of the registers to 128 -ta=maxregcount:128

• Memory is still not accessed contiguously, especially on the red-black kernels

17

• Red-black solver with 3 sweeps, CFL 0.1

• Linear scaling with number of iterations once data movement cost is offset

FINAL SOLUTION TIMES

18

• Red-black solver with 3 sweeps, CFL 0.1

• Linear scaling with grid size once data movement cost is offset

FINAL SOLUTION TIMES

19

• Red-black solver with 3 sweeps, CFL 0.1

• Best speedup of 49 for a large enough grid and number of iterations

FINAL SPEEDUP

20

CONVERGENCE OF THE LINEAR SOLVERS (1 OF 2)

• 161 x 2 x 65 mesh, convergence to 10−11Same run times

21

CONVERGENCE OF THE LINEAR SOLVERS (2 OF 2)

• 161 x 31 x 65 mesh, convergence to 10−11

22

EFFECTIVE SPEEDUP

• 161 x 31 x 65 mesh, convergence to 10−11

GPU - Red-black solver CPU - Red-black solver CPU – SSOR solver

109.3 sec 4329.6 sec 3140.0 sec

• Speedup of 39 compared to the same solver on CPU• Speedup of 29 compared to the SSOR scheme on CPU

23

The effective speedup is the same as speedup in 2D, and lower but still good in 3D!

• Conclusions

• A CFD solver has been ported to GPU using OpenACC

• Speedup on the order of 50 X compared to a single CPU core

• Red-black solver replaced the LU-SSOR solver with little to no loss of performance

• Future work

• Further optimization of data transfers and loops

• Extension to MPI

CONCLUSIONS AND FUTURE WORK

24

• Oak Ridge National Lab

• Organizing and letting us participate in the 2017 GPU Hackathon

• Providing access to Power 8 and P100 GPUs on SummitDev

• NVIDIA

• Providing access to P100 GPUs on the psg cluster

• Everyone else who helped with this work

ACKNOWLEDGEMENTS

25

• Contact

• Nicholson K. Koukpaizan

• nicholsonkonrad.koukpaizan@gatech.edu

• Please, remember to give feedback on this session

• Question?

CLOSING REMARKS

26

Nonlinear Computational Aeroelasticity Lab

BACKUP SLIDES

27

• Navier-Stokes equations

𝜕

𝜕𝑡නΩ

𝑼𝑑𝑉 + ර𝜕Ω

(𝑭𝑐−𝑭𝑉)𝑑𝑆 = 0

𝑼 = 𝜌 𝜌𝑢 𝜌𝑣 𝜌𝑤 𝜌𝐸 𝑇

• 𝑭𝐶 , inviscid flux vector, including mesh motion if needed (Arbitrary Lagrangian-Euler formulation)

• 𝑭𝑉 , viscous flux vector

• Loosely coupled turbulence model equations added as needed

• Laminar flows only in this work

• Addition of turbulence does not change the GPU performance of the application

GOVERNING EQUATIONS

28

• Explicit treatment of fluxes

• 2nd order central differences with 4th order Jameson dissipation

• Implicit treatment of fluxes

• Steger and Warming flux splitting

• Dual time stepping, with 2nd order backward difference formulation

• Form of the final equation to solve

Ω𝑖𝑗𝑘

Δ𝑡

Δ𝑡

Δ𝜏+3

2𝑰 +

𝜕𝑹

𝜕𝑼

𝑚

Δ𝑼𝒎 = −𝑹𝑚 −Ω𝑖𝑗𝑘

Δ𝑡

3𝑼𝑚 − 4𝑼𝑛 + 𝑼𝑛−1

2

• Need a linear solver!

DISCRETIZED EQUATIONS

29