University of Michigan XFlow GroupCorrespondence: Marco Ceze, mceze@umich.edu PI: Krzysztof Fidkowski

C3.2 Turbulent Flow over the DPW III Wing Alone Case

1. Code description

XFlow is a high-order discontinuous Galerkin (DG) finite element solver written in ANSI C, intended tobe run on Linux-type platforms. Relevant supported equation sets include compressible Euler, Navier-Stokes, and RANS with the Spalart-Allmaras model. High-order is achieved compactly within elementsusing various high-order bases on triangles, tetrahedra, quadrilaterals, and hexahedra. Parallel runs aresupported using domain partitioning and MPI communication. Visual post-processing is performedwith an in-house plotter and with TecPlot. Output-based hp-adaptivity is available using discreteadjoints.

2. Case summary

Convergence to steady state on the initial mesh (Figure 3(a)) at p = 1 was achieved using the con-strained pseudo-transient continuation method [3] directly at the nominal flow conditions. The hp-adapted runs used the method presented in [1].

Runs were performed on the U.S. Department of Defense’s Diamond supercomputer using 400 (hprun) and 600 (uniform h and isotropic h runs) cores. On one core of U.S. D.O.D.’s Diamond machine,one TauBench unit is equivalent to 8.55 seconds of compute time.

The CPU time expressed in terms of work units in the figures below is cumulative and, hence, itaccounts for the total time taken from the beginning of the calculation. These time stamps includethe time taken for the primal and dual solves, for error estimation, and for mesh adaptation.

3. Meshes

The initial high-order curved mesh was generated by first creating a multiblock linear mesh usingICEM CFD (with the geometry provided on the workshop website), and then agglomerating 3 × 3blocks of linear cells into q = 3 high-order elements. This mesh is the same as provided in the HOworkshop’s website.

4. Results

Figure 1 shows the convergence history for the runs.Note, in Figure 1, that the isotropic-h adaptation and uniform-h refinement runs used fewer it-

erations than the hp-adaptation run for the first solution. The first solution is the same for all runsbut the isotropic-h and uniform-h runs used an updated line-search algorithm that is approximatelytwice as efficient as the former solution update method. We acknowledge that all runs should havebeen performed with the same solver setup but we did not have enough time to perform the hp runwith the updated solver.

We use the Spalart-Allmaras turbulence model with Oliver’s [4] modifications in which the flow isassumed to be fully turbulent. The discrete SA equation is scaled according to reference [2]. Also,Persson and Peraire’s [5] shock-capturing method is used to improve stability. At each adaptivestep, fadapt = 10% of the elements with largest contributions to the error estimate are selected forrefinement.

2013 High-Order CFD Workshop 1 University of Michigan XFlow Group

0 500 1000 1500 2000 2500 3000 350010

−4

10−2

100

102

104

106

108

Nonlinear iteration

|R|

Anisotrpic hp

Isotropic h (p=1)

Uniform h (p=1)

Figure 1: Residual convergence history for all runs.

Figure 2 shows the results organized as requested in the case description. No truth values for liftand drag are available for this case, hence, we assess output convergence by examining the “flatness”of the convergence curves. For the output-adapted runs, the output can be corrected by its errorestimate – dashed lines in Figures 2(a) and 2(b).

Figure 3 shows the initial and improved meshes with contours of pressure and turbulence modelworking variable. Note in Figure 3(d) that the higher order elements are mostly located before andafter the shock and in the turbulent boundary layer. Anisotropic h refinement is performed at theshock region, at shock-boundary-layer interaction, on the wake and along the stagnation streamlines.

Table 1 shows the fraction of choice for each refinement option for the hp run. Note that, apartfrom the first adaptive step, p-refinement is generally chosen for ∼ 1% of the elements selected forrefinement. However there seems to be a increasing trend of p-refinement towards the end of theadaptive process.

Table 1: DPW Wing 1, M∞ = 0.76, α = 0.5o, Re = 5 × 106, drag-based adaptation: fraction ofchoice for each refinement option; iso-h: isotropic h-refinement; sc-h: single-cut h-refinements; dc-h:double-cut h-refinements; iso-p: isotropic p-refinement.

Refinement options

Adaptive step iso-h sc-h dc-h iso-p

1 0.000 1.000 0.000 0.000

2 0.000 9.954 × 10−1 1.426 × 10−3 3.137 × 10−3

3 2.455 × 10−4 9.944 × 10−1 1.719 × 10−3 3.683 × 10−3

4 2.125 × 10−4 9.868 × 10−1 4.675 × 10−3 8.287 × 10−3

5 1.833 × 10−4 9.892 × 10−1 4.216 × 10−3 6.416 × 10−3

6 0.000 9.849 × 10−1 4.721 × 10−3 1.039 × 10−2

7 4.025 × 10−4 9.808 × 10−1 8.453 × 10−3 1.033 × 10−2

8 1.127 × 10−4 9.794 × 10−1 8.002 × 10−3 1.251 × 10−2

2013 High-Order CFD Workshop 2 University of Michigan XFlow Group

10−3

10−2

10−1

0.02

0.021

0.022

0.023

0.024

0.025

0.026

0.027

1/nDOF1/3

CD

Anisotropic hp

Isotropic h (p=1)

Uniform h (p=1)

(a) Drag coefficient evolution with respect to number ofdegrees of freedom

105

106

107

0.02

0.021

0.022

0.023

0.024

0.025

0.026

0.027

Workunits

CD

Anisotropic hpIsotropic h (p=1)Uniform h (p=1)

(b) Drag coefficient evolution with respect to CPU time

10−3

10−2

10−1

0.38

0.39

0.4

0.41

0.42

0.43

0.44

0.45

0.46

0.47

1/nDOF1/3

CL

Anisotropic hp

Isotropic h (p=1)

Uniform h (p=1)

(c) Lift coefficient evolution with respect to number ofdegrees of freedom

105

106

107

0.38

0.39

0.4

0.41

0.42

0.43

0.44

0.45

0.46

0.47

Workunits

CL

Anisotropic hpIsotropic h (p=1)Uniform h (p=1)

(d) Lift coefficient evolution with respect to CPU time

Figure 2: DPW3 W1, M∞ = 0.76, α = 0.5o, Re = 5 × 106, drag-based adaptation: dashed lines: dragvalues corrected by error estimate. Note: the uniform-h and isotropic-h runs used a more efficientsolution update method.

References

[1] Marco Ceze and Krzysztof J. Fidkowski. Anisotropic hp-adaptation framework for functionalprediction. AIAA Journal, 51(2):492–509, February 2013.

[2] Marco Ceze and Krzysztof J. Fidkowski. Drag prediction using adaptive discontinuous finiteelements. In 51st AIAA Aerospace Sciences Meeting and Exhibit, 2013.

[3] Marco Antonio de Barros Ceze. A Robust hp-Adaptation Method for Discontinuous Galerkin Dis-cretizations Applied to Aerodynamic Flows. PhD thesis, The University of Michigan, 2013.

[4] Todd A. Oliver. A High–order, Adaptive, Discontinuous Galerkin Finite Elemenet Method for theReynolds-Averaged Navier-Stokes Equations. PhD dissertation, Massachusetts Institute of Tech-

2013 High-Order CFD Workshop 3 University of Michigan XFlow Group

(a) Initial pressure and ρν contours (29310 cubic ele-ments, p = 1).

(b) Pressure and ρν contours on the 1st level of uniformh-refinement (234480 cubic elements, p = 1).

(c) Pressure and ρν contours on the 4th drag-adaptedmesh with isotropic h refinement (144821 cubic ele-ments).

(d) Pressure and ρν contours on the 9th drag-adaptedmesh using anisotropic hp refinement (105715 cubic ele-ments). Note the p-order distribution on the y-slices.

Figure 3: DPW Wing 1, M∞ = 0.76, α = 0.5o, Re = 5 × 106: Initial and drag-adapted meshes withpressure and ρν contours. Note the improved wake and shock resolutions for the adapted meshes.

nology, Cambridge, Massachusetts, 2008.

[5] P.-O. Persson and J. Peraire. Sub-cell shock capturing for discontinuous Galerkin methods. In44th AIAA Aerospace Sciences Meeting and Exhibit, number 2006-112, 2006.

2013 High-Order CFD Workshop 4 University of Michigan XFlow Group

Case 3.2: Turbulent Flow over the DPW III WingAlone Case

University of Michigan - XFlow

Department of Aerospace EngineeringUniversity of Michigan

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 1/22

XFlow Code

Discontinuous Galerkin spatial discretization.Physicality-constrained pseudo-transient continuation (CPTC)nonlinear solver with line-search for improving convergence.Exact Jacobian with element-line-Jacobi preconditioner andGMRES linear solver.Roe solver for inviscid flux and BR2 for viscous discretization.MPI parallelization.Node-edge weighted mesh partitioning.Support for curved meshes.Oliver/Allmaras modification to original SA turbulence model.Shock-capturing via element-wise constant artificial viscosity.Output-based anisotropic hp-adaptation.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 2/22

CPTC problem statementConsider the semi-discrete flow equations:

Ut = −R(U).

The discrete solution U is used to approximate the state, u, as afield uH,p(t , x). In finite elements:

uH,p(t , x) =∑

j

Uj(t)φH,pj (x).

The field uH,p(t , x) is subject to physical realizability constraints:

p(uH,p(t , x))

p∞> 0,

ρ(uH,p(t , x))

ρ∞> 0 and

ν(uH,p) + νt (uH,p)

ν(uH,p)> 0.

These realizability constraints are violated sometimes.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 3/22

Marching to steady-state

(M 1

∆t+∂R∂U

∣∣∣Uk

)︸ ︷︷ ︸

A

∆Uk = −R(Uk )

Pseudo-transient continuation

Multiply left-hand side by its transpose:

∆UT AT A∆U = −∆UT AT R(U)︸ ︷︷ ︸∂f∂U

≥ 0.

∆U is a descent direction for:

f (U) = |Rt (U)|2L2= |M 1

∆t(U− Uk ) + R(U)|2L2

.

No direct mechanism to avoidphysicality constraints!

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 4/22

Handling physicality constraints

We propose augmenting the residual with a penalty vector:

Rp(U) = R(U) + P(U).

For a repelling effect w.r.t the constraints, it is sufficient to makeP(U)T R(U) > 0. We choose:

P = Φ R,

where Φ is a diagonal matrix with elemental penalties:

PκH (UκH (t), µ) = µ

Ni∑i

Nj∑j

wj

ci(uH,p(t , xj)).

where µ is the penalty factor, ci are the constraints, and xj areinterrogation points.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 5/22

Handling physicality constraints

Apply PTC to Rp and the equation for the state update becomes:(I + Φ)−1M∆t︸ ︷︷ ︸

a

+∂R∂U

+ (I + Φ)−1(∂Φ

∂UR(U)

)︸ ︷︷ ︸

b

∆U = −R(U).

The elemental ∆t gets amplified by a factor (1 + PκH ).In the limit ∆t →∞, the solution seeks a minimum of |Rp|L2 .As the state in an element approaches a non-physical condition,term "a" vanishes locally while "b" remains due to the derivative ofthe inverse barrier function.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 6/22

Constrained PTC exampleSolve a simple nonlinear algebraic system:

sin(4πx1x2)− 2x2 − x1 = 0(4π − 1

4π

)(e2x1 − e) + 4ex2

2 − 2ex1 = 0

−1 ≤ x1 ≤ 1−1 ≤ x2 ≤ 1

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x2

x1

Prohibited

PTC

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x2

x1

Prohibited

Constrained PTCUofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 7/22

DPW 3 W1 - M∞ = 0.76, Re = 5× 106, α = 0.5◦, p = 1

Cubic mesh generated by agglomerating linear cells.y+ ≈ 1 based on a flat-plate correlation.804 CPU’s, 5 hours, maximum of 4k iterations.

Mesh and pressure contours (29310 elements) Mach number and ρν contours

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 8/22

Comparison for different solver parametersNo scaling for the discrete SA equation.Success of all runs:

→ converged,→ timeout or max iterations,→ CFL below minimum or non-physical.

PTC CPTCMPC LS LS+G MPC LS LS+G

EXPur run 1.1.1 run 1.2.1 run 1.3.1 run 2.1.1 run 2.2.1 run 2.3.1SER run 1.1.2 run 1.2.2 run 1.3.2 run 2.1.2 run 2.2.2 run 2.3.2RDM run 1.1.3 run 1.2.3 run 1.3.3 run 2.1.3 run 2.2.3 run 2.3.3

mRDM run 1.1.4 run 1.2.4 run 1.3.4 run 2.1.4 run 2.2.4 run 2.3.4

Performance of converged runs.

Run ID Nonlinear iterations GMRES iterations Wall time (seconds)2.1.1 1.000 (1255) 1.000 (68253) 1.000 (5.694× 103s)2.1.4 0.897 0.935 0.8452.2.1 0.880 0.944 1.0212.2.4 0.751 1.002 1.127

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 9/22

Weighted mesh partitioningThe mesh is represented as an irregular graph where elementsare nodes and interior faces are edges.

The sets in red represent lines of the line-Jacobi preconditioner.The inter-domain communication stores the data in one layer offictitious elements neighboring each inter-domain boundary.We use the k -way partitioning algorithm implemented inParMETIS.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 10/22

Weighted mesh partitioningNode weights are based on the number of non-zeros in theself-blocks of the residual Jacobian:

ωκH = (pκH + 1)2·dim.

The edge weights are computed in the following sequence:1 Loop through edges of the graph (faces of the mesh) and compute:

ω∂κH\∂D = (p+κH + 1)dim + (p−

κH + 1)dim

2 Loop through lines of the preconditioner and augment ω∂κH\∂Dbased on the valence (connections per node) vκH :

ω∂κH\∂D ⇐ ω∂κH\∂D ·max(v+κH , v−κH ).

Step 1 assigns weights proportional to the amount of data transferand step 2 increases the weight of connections between elementsthat are strongly coupled.

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 11/22

Is it worth doing this?...YES!

DPW III Wing 1 - M∞ = 0.76, Re = 5× 106, α = 0.5◦,hp-adaptation on 720 CPU’s.Challenge: what to do with empty partitions?...we lower the disparity of weights

Primal solution time

2 2.5 3 3.5 4 4.5x 10

5

0

0.5

1

1.5

2x 104

Number of degrees of freedom

Prim

also

lvetime(seconds)

cNZ

unweighted

cNZ

weighted

Adjoint solution time

2 2.5 3 3.5 4 4.5x 10

5

0

1000

2000

3000

4000

5000

Number of degrees of freedom

Adjoin

tso

lvetime(seconds)

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 12/22

Is it worth doing this?...YES!

DPW III Wing 1 - M∞ = 0.76, Re = 5× 106, α = 0.5◦,hp-adaptation on 720 CPU’s.Number of GMRES iterations is directly related to using thepreconditioner lines in the partitioning.

Adaptation time

2 2.5 3 3.5 4 4.5x 10

5

200

300

400

500

600

700

800

900

Number of degrees of freedom

Adapta

tion

time(seconds)

cNZ

unweighted

cNZ

weighted

GMRES iterations for primal and dual solves

2 2.5 3 3.5 4 4.5x 10

5

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8x 105

Number of degrees of freedom

NumberofGM

RES

iterations

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 13/22

Mesh Improvement MethodThe output error is approximated as:

δJ ≈ −∑

κH∈T H

RκH (uH,p,ψH,p+1 − ψH,p).

The adaptive indicator is defined as:

ηκH =∣∣∣RκH (uH,p,ψH,p+1 − ψH,p)

∣∣∣.ψH,p+1 is approximated by 15 lean-Jacobi smoothing iterations.f adapt = 10% of the elements with the highest ηκH are refined ateach step.A set of discrete refinement options is considered, e.g.:

pp

(a) x

p

p

(b) y

p

p p

p

(c) xy

p+1

(d) p

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 14/22

hp-AdaptationQuestion:Which direction should we refine?

Our proposition:Rank the refinement options based on a function:

m(i) =b(i)c(i)

c(i) is a measure of the computational cost of refinement option i(e.g. non-zeros in the Jacobian).b(i) measures the gain in accuracy due to the refinement option i(e.g. output sensitivity to residual perturbations).Balance between high-cost-low-error and low-cost-high-erroroptionsChoose the option with the highest m(i).

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 15/22

Residual ConvergenceFlow initialized with free stream conditions.CPTC + Line-search+EXPur.

0 500 1000 1500 2000 2500 3000 350010

−4

10−2

100

102

104

106

108

Nonlinear iteration

|R|

Anisotrpic hp

Isotropic h (p=1)

Uniform h (p=1)

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 16/22

Drag Convergence

Only drag-adapted runs.Dashed lines correspond to output correction by its error estimate.Runs used 400(hp) to 600 cores of US DOD’s Diamond machine(wu = 8.55s).

10−3

10−2

10−1

0.02

0.021

0.022

0.023

0.024

0.025

0.026

0.027

1/nDOF1/3

CD

Anisotropic hp

Isotropic h (p=1)

Uniform h (p=1)

Degrees of freedom

105

106

107

0.02

0.021

0.022

0.023

0.024

0.025

0.026

0.027

Workunits

CD

Anisotropic hpIsotropic h (p=1)Uniform h (p=1)

Workunits

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 17/22

Lift Convergence

10−3

10−2

10−1

0.38

0.39

0.4

0.41

0.42

0.43

0.44

0.45

0.46

0.47

1/nDOF1/3

CL

Anisotropic hp

Isotropic h (p=1)

Uniform h (p=1)

Degrees of freedom

105

106

107

0.38

0.39

0.4

0.41

0.42

0.43

0.44

0.45

0.46

0.47

WorkunitsC

L

Anisotropic hpIsotropic h (p=1)Uniform h (p=1)

Workunits

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 18/22

Drag-based hp adaptation statistics

Refinement optionsAdaptive step iso-h sc-h dc-h iso-p

1 0.000 1.000 0.000 0.0002 0.000 9.954× 10−1 1.426× 10−3 3.137× 10−3

3 2.455× 10−4 9.944× 10−1 1.719× 10−3 3.683× 10−3

4 2.125× 10−4 9.868× 10−1 4.675× 10−3 8.287× 10−3

5 1.833× 10−4 9.892× 10−1 4.216× 10−3 6.416× 10−3

6 0.000 9.849× 10−1 4.721× 10−3 1.039× 10−2

7 4.025× 10−4 9.808× 10−1 8.453× 10−3 1.033× 10−2

8 1.127× 10−4 9.794× 10−1 8.002× 10−3 1.251× 10−2

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 19/22

Initial and uniform-h meshes

Initial solution (p = 1) Uniform-h refinement (p = 1)

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 20/22

Initial and uniform-h meshes

Isotropic-h adaptation (p = 1) hp-Adaptation (p = 1− 3)

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 21/22

Conclusions

The physicality-constrained PTC made residual convergencestraight-forward.Line-search accelerates the convergence of Newton-basedsolvers.Node-edge-weighted partitioning improved the parallel efficiencyof hp runs.Can we get similar output convergence for cheaper?

UofM 2nd International Workshop on High-Order CFD Methods, May 27-28, Cologne, Germany C3.2 22/22