5

Final ALCC Report for 2017-2018 “Numerical

Simulation of Turbulent Flows in Advanced Steam

Generators – Year 3 (TurbFlowGen 2) Project”

Mathematics and Computer Science Division

September 28, 2018

About Argonne National Laboratory

Argonne is a U.S. Department of Energy laboratory managed by UChicago Argonne, LLC undercontract DE-AC02-06CH11357. The laboratory’s main facility is outside Chicago, at 9700 SouthCass Avenue, Lemont, Illinois 60439. For information about Argonne and its pioneering science andtechnology programs, see www.anl.gov.

Availability of This Report:

This report is available, at no cost, at http://www.osti.gov/bridge. It is also available on paper tothe U.S. Department of Energy and its contractors, for a processing fee, from:U.S. Department of EnergyOffice of Scientific and Technical InformationP.O. Box 62Oak Ridge, TN 37831-0062phone (865) 576-8401

fax (865) 576-5728

reports@adonis.osti.gov

Disclaimer

This report was prepared as an account of work sponsored by an agency of the United States Government.Neither the United States Government nor any agency thereof, nor UChicago Argonne, LLC, nor any of theiremployees or officers, makes any warranty, express or implied, or assumes any legal liability or responsibilityfor the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, orrepresents that its use would not infringe privately owned rights. Reference herein to any specific commercialproduct, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarilyconstitute or imply its endorsement, recommendation, or favoring by the United States Government orany agency thereof. The views and opinions of document authors expressed herein do not necessarily stateor reflect those of the United States Government or any agency thereof, Argonne National Laboratory, orUChicago Argonne, LLC.

Final ALCC Report for 2017-2018 “Numerical

Simulation of Turbulent Flows in Advanced Steam

Generators – Year 3 (TurbFlowGen 2) Project”

Aleksandr V. Obabko1, Vakhtang Makarashvili1, Elia Merzari2,1,Haomin Yuan2, Jonathan Lai3, and Paul F. Fischer4,1

1Mathematics and Computer Science Division, Argonne NationalLaboratory

2Nuclear Engineering Division, Argonne National Laboratory3Department of Nuclear Engineering,Texas A&M University

4Department of Mechanical Science and Engineering and Department ofComputer Science, University of Illinois at Urbana-Champaign

Mathematics and Computer Science DivisionArgonne National Laboratory

September 28, 2018

Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Helical Coil Steam Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 Complex Meshing Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 Spacer Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3.2 Methods: Nek5000, Conditions and Setup . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.3 Results for Balanced and Unbalanced Flow . . . . . . . . . . . . . . . . . . . . . . . . 16

3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4 Test Simulations of Other NEAMS Geometries. . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.1 Cold Leg Mixing Benchmark Experiment Configuration. . . . . . . . . . . . . . . . . . 20

4.2 Lightbridge Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

i

Executive Summary

The NEAMS program aims to develop an integrated multi physics simulation capability “pellet to plant” forthe design and analysis of future generations of nuclear power plants. In particular, the Reactor ProductLine code suite’s multi resolution hierarchy is being designed to ultimately span the full range of length andtime scales present in relevant reactor design and safety analyses, as well as scale from desktop to petaflopcomputing platforms. Flow-induced vibration (FIV) is widespread problem in energy systems because theyrely on fluid movement for energy conversion. Vibrating structures may be damaged as fatigue or wear occurs.In particular, nuclear fuel rods and steam generators have been known to suffer from FIV and related failures.Nuclear fuel assemblies are exposed to severe thermal, mechanical, and irradiation loads during operation.Global core and local fuel assembly flow fields often result in flow-induced fuel assembly and fuel rod vibration.When coupled with other factors, the vibration might result in excessive fretting wear of fuel rod cladding,and fuel assembly motions may even contribute to significant power oscillations. This phenomenon and itsconsequences on operational safety margins in fuel assembly designs are primary concerns for nuclear fueldesigners. Understanding the particular set of thermal, mechanical, flow, and radiation conditions to which afuel assembly is subjected and how the fuel assembly responds dynamically to those conditions are importantin assessing operational safety margins, and in determining the risk of excessive wear. The ability to bettercharacterize these factors will also enable further design exploration that can lead to better, safer, and moreeconomically viable designs.

Advanced modeling and simulation based on coupled structural and fluid simulations have the potentialto predict FIV in a variety of designs, reducing the need for expensive experimental programs, especiallyat the design stage. Over the past several years, the Reactor Product Line has developed the integratedmulti physics code suite SHARP. The goal of developing such a suite is to perform multi physics neutronics,thermal/fluid, and structural mechanics modeling of the components inside the full reactor core or portions ofit with a user-specified fidelity. In particular, SHARP contains high-fidelity single-physics codes for structuralmechanics (Diablo) and for fluid mechanics calculations (Nek5000). Both codes are state-of-the-art, highlyscalable tools that have been extensively validated. These tools form a strong basis on which to build an FIVmodeling capability.

In the present report we discuss part of the efforts in the past three years toward the development of thiscapability and its demonstration on a state-of-the-art fuel assembly experiment. We focus on the validationof the fluid dynamic simulation with Nek5000. The work is performed collaboratively with Framatome, whichprovided the experimental data for comparison. Numerical simulation results are compared with experimentalaverages and root mean squares of the velocity demonstrating that Nek5000 can reproduce the flow field inthis geometry with reasonable accuracy.

. . .

As a TH component of the SHARP multiphysics toolkit, the Nek5000 V&V-driven development aims atassessing the toolkit’s capability of addressing the high-fidelity modeling and simulation needs identified asimportant to both the DOE-NE ART program and the nuclear industry. This and last year’s verification andvalidation-driven usage of ALCC resources has focused on two benchmarks for the open-source highly-scalablespectral-element code Nek5000.

For the first benchmark, we have performed the 37-pin PLANDTL sodium experiment LES campaignas a part of the advanced M&S activity of CNWG under the U.S.-Japan Bilateral Commission Civil NuclearCooperation on advanced reactor R&D between DOE and JAEA. Run with the latest code release using upto 1 million MPI ranks, the conjugate heat transfer simulation in this benchmark geometry is the largestup-to-date not only with Nek5000 but with any CFD solver. Once our JAEA colleagues share the promisedexperiment data, we will validate the simulation results and this code setup for future uncertainty estimationand for benchmarking of lower-fidelity faster-turn-around modeling tools.

As a part of the second benchmark, SESAME INERI 61-pin cross-verification exercise, we extracted anddelivered Nek5000 LES data to our European Horizon 2020’s colleagues from NRG, SCK-CEN, UGent, and

ii

ENEA. Originally funded by NIC Cost-Share Award, the 61-pin FOA project was a joint venture with AREVA,TerraPower, and TAMU that was aimed at design and completion of well-defined first-of-a-kind validationcampaign for the TH CFD modeling of hexagonal packed fuel assemblies with helically wire-wrapped pingeometries typical of SFRs.

Our prior experiences with similar collaborations, including the previous INERI with NRG, SCK-CEN,ENEA, and UGent, have highlighted the importance of such interactions for the advancement of reliablecomputational tools and the expertise to employ the tools reliably. Furthermore, the collaboration withEurope and Japan provides a venue with NEAMS contributors to work closely with contributors to the ARTprogram, one of NEAMS primary customers within DOE-NE and the primary sponsor of SFR technologyR&D. And the ALCC resources were crucial for its successful continuation.

iii

1 Introduction

This and last year ALCC1 allocation was focused on V&V2 of Nek50003 as a TH4 component of the SHARP5

using two benchmarks. The experiment benchmark PLANDTL was conducted by JAEA6 using liquid sodium.The second benchmark is the cross-verification exercise with our partners from SESAME7 INERI8 projectbetween the U.S. DOE and Euratom.

Following multiyear efforts for development of coupled neutronics, CFD,9 and CSM10 analysis capabilities,the SHARP multi-physics toolkit was released in March 2016. In following two years, the main focus ofthe RPL11 shifted to testing, validation and verification of the SHARP component codes and multiphysicstoolkit in applications to challenging problems for advanced reactor designs. The objective is to assessSHARP capabilities in addressing the high-fidelity modeling and simulation needs that have been identified asimportant to both the DOE-NE ART12 program and the nuclear industry. In addition to ongoing developmentof the advanced reactor SAM,13 a portion of the FY18 efforts under RPL continues to support further modelingimprovements for each of the key components of the SHARP toolkit: the neutronics module PROTEUS, theCFD module Nek5000, and the CSM module DIABLO. Focusing here primarily on Nek5000, the open-sourcehighly scalable code based on SEM,14 we strive to test and improve the predictive and validated capability ofproviding accurate reference solution (typically LES15) that could be further used for benchmarking andbetter uncertainty estimation of lower-fidelity faster-turn-around approaches during parameter scoping anddesign modifications.

For the first wire-wrap validation benchmark, we report the largest-to-date LES results in the 37-pinPLANDTL16 sodium experiment geometry that is a part of the advanced M&S17 activity of CNWG18 underthe U.S.-Japan Bilateral Commission Civil Nuclear Cooperation on advanced reactor R&D between theU.S. Department of Energy and JAEA. Using test data from PLANDTL, we will validate SHARP and SAMmodels for detailed thermal-hydraulic analysis of flow in wire-wrapped SFR rod bundle configurations underprototypical operating conditions. This year we have focused on conjugate head transfer LES after carefulpreparation of computational mesh and isothermal initial conditions in axially periodic geometry in order tooptimize the CPU and funding usage. Note that we have performed all simulations and data processing withthe most recent release of Nek5000, i.e. version 17.

The second benchmark reported here is SESAME INERI 61-pin cross-verification exercise with our

1Argonne Leadership Computational Challenge2Verification and Validation3https://nek5000.mcs.anl.gov4thermal-hydraulics5Simulation-based High-efficiency Advanced Reactor Prototyping6Japan Atomic Energy Agency7Simulations and Experiments for the Safety Assessment of MEtal Cooled Reactors8Nuclear Energy Research Initiative9Computational Fluid Dynamics

10Computational Structural Mechanics11Reactor Product Line12Advanced Reactor Technologies13System Analysis Module14spectral-element method15large-eddy simulation16PLANt Dynamic Test Loop17Modeling and Simulation18Civil Nuclear Energy Research and Development Working Group

1

European Horizon 2020’s colleagues from NRG,19 SCK-CEN,20 UGent,21 and ENEA.22 Originally fundedby NIC23 Cost-Share Award, the 61-pin FOA project was a joint venture with AREVA, TerraPower, andTAMU24 that was aimed at design and completion of well-defined first-of-a-kind validation campaign forthe TH25 CFD modeling of hexagonal packed fuel assemblies with helically wire-wrapped pin geometriestypical of SFRs.26 Thermal-hydraulic analysis of flows in the wire-wrapped rod bundles, where wires arehelically wrapped around the pin to serve simultaneously as a spacer and enhancer of thermal mixing, ischallenging for several reasons. The flow typically has a large Reynolds number and thus is fully turbulent.The wire wrap introduces additional turbulence and establishes a bulk rotation of the fluid within the assembly.Because it creates multiple contact points and lines, the wire wrap makes the geometry and, consequently, thecomputational mesh relatively complex and large. And since the reactor comprises several hundred assemblies,most thermal-hydraulic analysis to date has been based on coarse lumped-parameter models. More recently,high-performance workstations have enabled the use of subchannel models in which a few hundred degrees offreedom per channel are used to represent mass, momentum, and energy balances at axial positions along thechannel and where interchannel mixing is modeled using experimentally determined correlations. Becausesubchannel models require far less memory and execute relatively quickly compared with CFD methods, theywill likely continue to play an important role in reactor-scale analysis for the foreseeable future. However,they are not suitable for evaluating thermal-hydraulic performance of a given assembly in the presence oflarge deformations such as those encountered toward the end of life, which was the original focus of the 61-pinFOA benchmark. In this report we provide describe the data that have been extracted and delivered to ourSESAME INERI partners. In exchange for the Nek5000 61-pin LES our European colleagues will providerare transient integral system response from SFR data involving the Phenix Dissymmetric Test, performedby the French CEA,27 with measurements of the reactor systems transient during a controlled loss-of-coolantfrom one coolant loop.

Our prior experiences with similar collaborations, including the previous INERI with NRG, SCK-CEN,ENEA, and UGent, have highlighted the importance of such interactions for the advancement of reliablecomputational tools and the expertise to employ the tools reliably. Furthermore, the collaboration withEurope and Japan provides a venue with NEAMS contributors to work closely with contributors to the ARTprogram, one of NEAMS primary customers within DOE-NE and the primary sponsor of SFR technologyR&D. And the ALCC resources were crucial for its successful continuation.

2 Helical Coil Steam Generators

This year we continued the simulation campaign of helical coil steam generators. First, we have extended ofprevious one-way coupled calculations performed with Nek5000 and Diablo aimed at conducting validationand simulating available FIV experiments in helical steam generators in the turbulent buffeting regime. Itwas demonstrated that in the buffeting regime one-way coupling was judged sufficient because the pressureloads do not cause substantial displacements. However, higher speeds led to a reduction in accuracy and theneed for a fully coupled capability for velocity or approaching the critical velocity and fluid elastic instability.In this report we demonstrate such fully coupled capability on the Argonne test case discussed in previousyear allocation reports.

19Nuclear Research and Consultancy Group20Belgian Nuclear Research Centre21University of Gent22Italian National Agency for New Technologies, Energy and Sustainable Economic Development23Nuclear Infrastructure Council24Texas A&M University25Thermal-Hydraulic26Sodium cooled Fast Reactors27Alternative Energies and Atomic Energy Commission

2

Figure 1: Magnified by a factor of 104 times for visualization, HCSG tube (quasi-static) displacementcomputed in Diablo for pressure field solution of Nek5000.

In the recent development, we have continued implementation of a two-way coupling scheme in SHARPto combine Nek5000 CFD and Diablo CSM codes. This is achieved by a two-way coupling: Nek5000 transferpressure distribution on structure surface to Diablo, and Diablo transfer computed mesh displacement andmesh velocity to Nek5000. However, due to the size of the problem, (about 1 million elements in Nek5000),the two-way coupling is very slow on the debuging computational resource, i.e. LCRC machines Blues andBebop. A substantial progress has been made on installing complete SHARP package on BG/Q system.All third party libraries have been successfully compiled prior to SHARP package installation and the testproblem Turek runs fine. Before a complete two-way coupling runs, a quasi-static calculation is run first tolet Nek5000 mesh deform to the quasi-static state with pressure loading on structure surface (Figure 1).

2.1 Complex Meshing Improvements

Here we report on improvement of meshing strategy that leads to a faster-turn-around of reference LESsimulations in complex geometries including HCSG and pebble bed geometry in addition to spacer grids(Section 3). The recently developed tet-to-hex meshing workflow where any CFD geometry is easily brokeninto tets with a subsequent split of every tet into 4 hexs followed by projection to the domain boundaries, hasbeen tested here and its LES is done with a reasonable count of pressure iterations which is a dominant cost.

3

Figures 2–4 show actual geometry of HCSG, a slice of velocity and zoom-in of mesh layout, correspondingly.Note that the flow is downward in Figures 3 and 4. Same meshing strategy has being applied to random

Figure 2: Geometry of helical coil steam generator flow with columns of tubes rotating in differencedirection.

pebble configuration ubiquitous in many Generation-IV (GenIV) designs of advanced nuclear reactors. As inHCSG project, we successfully tested the SEM mesh built with boundary layers needed for efficient simulationof momentum and heat transfer between the coolant and pebbles. Figures 5–7 show geometry of randompebble configuration, a slice of the LES flow, and its close-up with mesh, respectively. Note that the flowdirection is upward in Figures 6–7.

3 Spacer Grid

During the final year of the allocation, the focus has somewhat shifted to another geometry of interest to ourindustrial partners.

3.1 Introduction

Nuclear fuel assemblies are exposed to severe thermal, mechanical, and irradiation loads during operation.Global core and local fuel assembly flow fields often result in flow-induced vibration (FIV) within fuelassemblies. The cost of inadequate prediction of FIV phenomena within fuel assemblies can be high, sinceFIV may cause fuel failure and require extensive unit downtime. Hence, the study of FIV is of interest tonearly all vendors and for nearly all reactor designs. While empirical design methods and experience relatedto FIV might be adequately developed for typical geometries, design methods and experience related to FIVare far less developed for advanced fuel designs such as those planned for advanced reactors or for radicalnew grid spacer designs. As a result, numerical simulation or analytical prediction of FIV is increasinglyimportant.

The focus of this work is the development of a high-fidelity, finite-element analysis/computational fluid

4

Figure 3: A slice of helical coil steam generator flow simulated with Nek5000.

Figure 4: A close-up of helical coil steam generator flow simulated with Nek5000.

dynamics (FEA/CFD) capability for the simulation of FIV. An advanced numerical simulation capability formodeling FIV phenomena will help improve the analysis and evaluation of different design variants in terms ofvibrations and heat transfer performance. Such a capability will complement expensive experimental tests andreduce their cost, while providing a better understanding of the physics behind FIV. Work performed in theprevious three years led to the development of such a tool in the multi physics suite SHARP. One-way coupledcalculations have been successfully performed for helical steam generators with this tool using Nek5000 andDiablo, the CFD and structural mechanics components of SHARP, respectively [1, 2].

5

Figure 5: Geometry of a random pebble bed..

Figure 6: Random pebble bed flow simulated in Nek5000.

We describe here the simulation of a state-of-the-art fuel assembly experiment performed by Framatome,designed to assess fluid loads on a 5 by 5 rod bundle with grid spacers at various inlet conditions. We focuson the CFD side, since the fluid loads represent the highest uncertainty in the problem and they can beconsidered as an input to structural calculations in the turbulent fretting regime. Simulation runs wereperformed on Argonne’s supercomputer Mira, an IBM Blue Gene/Q architecture with 49,152 nodes (16 coresper node) at the Argonne Leadership Computing Facility.

Two cases of flow geometry were simulated with Nek5000 [3]: balanced and unbalanced. In the balancedflow the coolant enters the fuel rod assembly through 121 uniformly spaced inlet holes arranged in an 11x11matrix. The unbalanced case, on the other hand, features 14 larger holes placed on only one side of thehorizontal plane. The simulations were performed in three phases. The initial phase involved testing andadjusting both meshes while decreasing the viscosity to match the target Reynolds number of the physicalflow. The second phase involved running simulations long enough to achieve steady-state turbulent flow

6

Figure 7: Close-up of a random pebble bed flow simulated in Nek5000.

throughout the entire domain. The third phase involved production runs that collected statistics of threevelocity components and pressure. The results were plotted by using the visualization software VisIt, andpost processing and analyses were performed in Matlab.

We present here a detailed comparison between the numerical results and experimental data. Theresults show reasonable agreement between simulations and experiment, representing an important steptoward validation of Nek5000 as a simulation capability for FIV studies.

3.2 Methods: Nek5000, Conditions and Setup

The open-source fluid/thermal simulation code Nek5000 is designed specifically for transitional and turbulentflows in complex domains. Nek5000 is based on the spectral element method (SEM) [4], a high-order weightedresidual technique that combines the geometric flexibility of finite elements with the rapid convergence andtensor-product efficiencies of global spectral methods. Globally, the SEM is based on a decomposition ofthe domain into E smaller subdomains (elements), which are assumed to be curvilinear hexahedra (bricks)that conform to the domain boundaries. Locally, functions within each element are expanded as N th-orderpolynomials cast in tensor-product form, which allows differential operators on N3 gridpoints per element tobe evaluated with only O(N4) work and O(N3) storage.

The principal advantage of the spectral-element method is that convergence is exponential in N , whichimplies that significantly fewer gridpoints per wavelength are required to accurately propagate a signal (orturbulent structure) over the extended times associated with high Reynolds number flow simulations. Ahigh-order code involves slightly more work per gridpoint but not more memory access. We emphasizethat the reduction in the number of gridpoints is a function of the discretization choice, independent of theimplementation; the savings is realized by having an efficient SEM code that does not lead to increased costper gridpoint.

In addition to its high-order foundation, Nek5000 has several other features. It supports two formulationsfor spatial discretization. The first is the PN − PN−2 method [5, 6, 7], in which velocity and pressure spacesare represented as tensor-product polynomials of degree N and N − 2, respectively, in the reference elementΩ := [−1, 1]d, where d = 2 or 3. E elements of the spatial domain consists of the union of elements Ωe,which are parametrically mapped from Ω to yield a body-conforming mesh. The second is the low-Machnumber formulation [8, 9], which uses N -order approximation spaces for both the velocity and pressure.

7

Temporal discretization is based on a high-order splitting that is third-order accurate in time and reduces thecoupled velocity-pressure Stokes problem to four independent elliptic solves per timestep: one for each velocitycomponent and one for the pressure. The velocity problems are diagonally dominant and thus easily solvedby using Jacobi-preconditioned conjugate gradient iteration. The pressure substep requires a Poisson solveat each step, which is effected through multigrid-preconditioned GMRES iteration coupled with temporalprojection to find an optimal initial guess. Particularly important components of Nek5000 are its scalablecoarse-grid solvers that are central to parallel multigrid. The code features a fast direct solver that is optimalup to processor counts of P ≈ 104 and fast algebraic multigrid (AMG) for P ≈ 105 and beyond. Counts of 15GMRES iterations per timestep for billion-gridpoint problems are typical with the current pressure solver.

Nek5000 scales extremely well on the IBM Blue Gene/P and Blue Gene/Q architectures. The coderealizes excellent strong scaling, sustaining 60% parallel efficiency with as few as 2,000 points per process forP = 1, 048, 576. Typically, production runs have 5,000 - 10,000 points per process and thus run at higherparallel efficiencies.

The approach used to simulate turbulence in the present work is Large Eddie Simulation (LES), with anexplicit filter that mimics the deconvolution method [10]. In LES, large-scale turbulence is simulated whilesmaller scales are modeled. Since smaller scales have a nearly universal behaviour, LES is a more reliablemethodology than is Reynolds-averaged Navier-Stokes, in the sense that it generally depends less on themodelling assumptions. Nek5000 has been validated extensively both in Direct Numerical Simulation (DNS)and LES mode [11, 12].

The experimental geometry consists of a square prismatic subsection that includes the 5x5 spacer gridand fuel rods. The coolant flow comes from the bottom through numerous holes located below the fuelrods. Two configurations of the flow are considered: (1) balanced flow, where the coolant comes through 121uniformly spaced holes distributed in an 11x11 matrix, and (2) unbalanced flow, where the fluid flows through14 larger holes located only one side of the horizontal plane. Three-dimensional models of the balanced andunbalanced flow configurations are given in Figures 8 and 9, respectively.

Figure 8: 3D model for the balanced flow geometry.

The orientation of the spacer grid relative to the flow is a critical component of the validation processbecause of the asymmetric design features. We note that the coordinate system used in the CAD model and

8

Figure 9: 3D model for the unbalanced flow configuration.

subsequently in the Nek5000 simulations (referred to here as “new” - XY Z) is different from the coordinatesystem adopted in the experimental description document (“old” - X ′Y ′Z ′). We use only the new coordinatesystem throughout this report. One needs to apply the following transformation rule to go from the oldsystem to the new.

X ′ → Z (1)

Y ′ → −Y (2)

Z ′ → −X (3)

A 2D slice of the spacer grid in the new coordinate system is shown in Figure 10. This Figure makesclear the orientation of the grid in the new system and reveals some of the most complicated asymmetricfeatures of this geometry.

Experimental data were collected at four different elevations measured from the bottom (Zmin) forthe points defined above. Both flow geometries for the CFD simulations were truncated in order to makecomputations feasible and included only a small region beyond the last experimental plane. Two elevationswere below the spacer grid and two above. These locations are given in Table 1.

Nek5000 simulations were carried out in a nondimensional form using the reference length scale L and thereference velocity scale U to rescale the length, velocity and time using the following simple transformations:

l∗ =l

L(4)

u∗ =u

U(5)

t∗ =l∗

u∗=

t

L/U(6)

where l∗, u∗ and t∗ are nondimensional length, velocity, and time, respectively.

9

Figure 10: Top view of the spacer grid - XY plane depicting a 5x5 rod bundle matrix and intricategeometry details between the rods.

Location Z coordinate[% of Height]

H1 33.5H2 42.5N1 69.5N2 90.5

Table 1: Numerical results of the flow history recorded at these four elevations measured from thebottom (Zmin) and presented as a percentage of the model height.

We chose the rod diameter as the reference length scale. The velocity reference scale, on the other hand,was set to the inlet velocity that would result in a mean flow velocity of 2 m/s. The Reynolds number of theflow based on the rod diameter and the inlet velocity was then found to be

Re =LU

ν≈ 15, 000, (7)

where ν denotes the kinematic viscosity (dynamic viscosity divided by density - µ/ρ).

By far the most challenging and time-consuming part of this simulation project was building a workingcomputational mesh for both the balanced and unbalanced flow cases. The project went through multipleiterations of mesh building, testing, identifying problematic regions, and adjusting and rebuilding phases.Almost all the complications were caused by the spacer grid region, where many small and intricate geometricfeatures made building a hexahedral (hex) mesh from scratch infeasible. Were it not for the spacer gridregion, the rest of the geometry would have been trivial to mesh. A successful and practical meshing strategyturned out to be using the automatic tetrahedral (tet) meshing capability of ANSYS ICEM software and thenconverting a tet mesh into a hex mesh by splitting each tet element into four hex elements. The geometry was

10

first meshed between z ∈ [−10, 2.5] and then extruded below z = −10 and above z = 2.5. The hardest partof this process was making sure that the initial tet mesh was properly projected to the CAD geometry andthen adding thin boundary layers around the fuel pins and the outer walls throughout the entire geometry toensure the accuracy of the results in those areas.

Figure 11: Vertical XZ profiles of the final computational meshes at Y = 0. Left: balanced flowgeometry; right: unbalanced flow geometry.

Figure 11 shows XZ vertical slices (at Y = 0) of the final mesh for both the balanced and unbalancedflow geometries. The CAD geometries for both cases were truncated at Z = −11 and Z = 9 in order totransport transient flow features through the entire geometry in a reasonable time, thus making simulationscomputationally feasible. The mesh for the balanced flow configuration consisted of E = 3, 415, 283 elements,

11

while the unbalanced flow mesh had E = 2, 084, 165 elements.

Figure 12 is a close-up version of the same meshes shown in Figure 11, revealing mesh features abovethe inlet holes for both meshes. One can see from these snapshots how the initial, automatically generatedtet elements were later split into hex elements. The figure also shows high element densities around the fuelrods, ensuring adequate resolution in the critical regions of the flow.

Figure 12: Close-up vertical snapshots of the computational mesh above the inlet holes. XZ planeat Y = 0. Top: balanced flow configuration; bottom: unbalanced flow configuration.

Figure 13 shows vertical and horizontal slices of the mesh in the most complicated region of the geometry,the spacer grid. The small and intricate geometric features made building a hex mesh for this region and

12

Figure 13: Mesh in the spacer grid region, which is nearly identical for both flow geometries. Top:XZ slice at Y = 0; bottom: XY slice at Z = 0.

combining it with the rest of the geometry infeasible. The springs and spacers themselves not have theboundary layers, but the mesh elements around them are small enough to give a smooth solution and avoidnumerical instabilities.

Figure 14 illustrates a horizontal XY plane of the mesh above the spacer grid. Both flow configurationshave almost identical meshes in this portion of the geometry as well. The top part of Figure 14 shows the

13

Figure 14: Horizontal XY profile of the balanced geometry mesh above the spacer grid region. Top:entire XY plane; bottom: mesh around the B4 fuel rod.

14

Figure 15: XY mesh profiles at the inlet - Z = −8. Top: mesh for the 11x11 inlet holes for thebalanced flow; bottom: 14 larger inlet holes of the unbalanced geometry.

15

entire 5x5 fuel rod assembly, while the bottom part demonstrates the mesh around a single rod, including anice thin boundary layer around the rod; this ensures that the results close to the rod surfaces are correctand free of runaway numerical instability problems. The mesh also includes the boundary layers around thesquare walls of the geometry. Figure 15 shows a comparison between the balanced and unbalanced meshesin the inlet region. These are XY profiles at z = −8. The top picture is the balanced mesh of the 11x11inlet holes, while the bottom picture is the mesh of the 14 larger holes on the left side of the unbalanced flowgeometry.

After the mesh design was finalized, the next phase of the simulation project was to run both caseslong enough to reach a statistically steady turbulent state at the target Reynolds number (see Eq. 9). Thebalanced case was run to t = 7.65, while the unbalanced case was run to t = 3.73 in convective time unitsbefore acquiring data. All simulations were performed with polynomial order N = 7, corresponding to a totalnumber of mesh points n ≈ EN3 ≈ 1.17 billion and n ≈ 0.71 billion for the balanced and unbalanced flowmeshes, respectively. All the runs were done with a PN −PN−2 method for spatial discratization. Throughoutthe entire work, the approach to simulate turbulence was large eddie simulation, with an explicit filter thatmimics the deconvolution method. The filtering for the deconvolution LES mode was set to 5% on thelast mode for both the balanced and unbalanced flow simulations. The initial conditions at t = 0 were setto (vx, vy, vz) = (0, 0, 1). All subsequent runs were restarted from the velocity field profiles obtained bythe previous simulation. The boundary conditions were the following for both flow configurations: inletboundary at z = −11 with (vx, vy, vz) = (0, 0, 1), turbulent outflow at z = 9 and wall boundary conditions atthe outer walls and rod surfaces. The timestep was ∆t = 2.5 · 10−5 in convective units, or about 2 · 10−7

seconds for the balanced flow and ∆t = 1.25 · 10−5 in convective units, or about 1 · 10−7 seconds for theunbalanced flow. We used the characteristics time stepping scheme [13] for both flow configurations. Thisscheme avoids the CFL constraint in the semi implicit formulation by treating the non-linear convective termin Navier-Stokes as a material derivative which represents the change along the path line (characteristic)associated with the convecting velocity field. This allowed larger timesteps with the CFL in the region of 5-7for both simulations. The data were collected over the time interval of t ∈ [7.65, 11.6] (convective units) andt ∈ [3.73, 4.8] for balanced and unbalanced flow simulations, respectively. All the runs were performed onALCF’s supercomputer Mira - an IBM BlueGene/Q system using 8192 compute nodes with 32 MPI ranksper node, totalling 262144 MPI ranks for each simulation. This allowed excellent strong scaling, sustainingmore than 60%-70% parallel efficiency with around 4400 points per process for the balanced case and around2700 points per process for the unbalanced flow.

3.3 Results for Balanced and Unbalanced Flow

We present here the results for both the balanced and unbalanced flow simulations. The results for thebalanced case are based on the data collected over four convective time units (corresponding to about 33 msof physical time) and they exhibit satisfactory statistical convergence. The data for the unbalanced case,on the other hand, were collected only over one convective time unit (≈ 8.25 ms) and should be treated aspreliminary. The results can be significantly improved by running the simulation longer in order to collectmore statistics. Also, higher accuracy for both cases can be achieved by increasing the polynomial order oradjusting the mesh at certain places that would allow smoother profiles. We note that mesh tuning is a morefeasible approach since the increase of polynomial order often proves to be computationally too expensive formeshes of this scale. The next two subsubsections summarize the major results for each flow configuration.

First, we show some of the profiles of the instantaneous axial velocity, w. Figure 16 is a vertical XZprofile of the flow field, while Figures 17 and 18 are two horizontal XY profiles at H2 (below the spacergrid) and N1 (above the spacer grid), respectively. These plots show the main turbulent features of theaxial flow. Figure 16 shows the highly turbulent flow coming through the inlet holes and entering the rodassembly region along with convective acceleration of the flow in the spacer grid. Higher flow velocities canbe observed in channels between the rods in Figures 17 and 18, both demonstrating very turbulent axialflows. Time-averaged axial velocity fields for planes N1 and N2 above the spacer grid are given in Figures 19

16

Figure 16: Vertical XZ profile of the instantaneous axial velocity flow field at y = 0.

17

Figure 17: Horizontal XY profile of the instantaneous axial velocity flow field at H2.

Figure 18: Horizontal XY profile of the instantaneous axial velocity flow field at N1.

18

Figure 19: Horizontal XY profile of the time averaged axial velocity flow field at N1.

Figure 20: Horizontal XY profile of the time-averaged axial velocity flow field at N2.

19

and 20, respectively. These smooth profiles provide a confirmation of the statistical convergence and reflectthe geometric characteristics of the grid. We point out that the color schemes are different on each plot. Theblue and the dark red are defined based on the minimum and maximum of each profile, respectively.

For the unbalance flow geometry, 2D profiles of the instantaneous axial velocity - w are presented here.Figure 21 is a vertical XZ profile of the axial flow, while Figures 22 and 23 are two horizontal XY profiles atH1 and N1, respectively. These plots are also a good demonstration of the main turbulent characteristics ofthe axial flow. Figure 21 clearly shows the unbalanced (i.e., nonuniform across XY ) turbulent flow comingthrough the one-sided inlet holes on the left and entering the rod assembly region along with the recirculationregion on the right side of the central rod. Figure 22 shows uneven axial velocity distributions in the channelsbetween the rods below the spacer grid at H1, while Figure 23 shows a similar plot above the spacer grid atN1. Again, the color schemes are different on each plot.

3.4 Conclusions

Nek5000 simulations were performed to model a turbulent coolant flow in a 5x5 fuel rod assembly for twoflow configurations — balanced and unbalanced. The results were compared with the experimental data.The data for the balanced case were collected over 4 convective time units (33 ms of physical time), whichallowed sufficient accumulation of statistics.

The unbalanced case, on the other hand, is preliminary and lacks sufficient statistical convergence. Itneeds longer run times to generate more reliable data. The mesh for this case can also be improved, especiallybetween the inlet holes and the fuel rods, in order to produce smoother velocity profiles.

Overall, the results presented here represent an important stepping stone toward validation of theNek5000 for simulating of the flow in fuel assemblies with grid spacers and the evaluation of fluid loads forFIV calculations. Given the complexity of the calculations and the flow involved, the work reported hererepresents one of the major milestones of the FIV work performed under the NEAMS program in the pastthree years.

4 Test Simulations of Other NEAMS Geometries.

In addition to core geometries simulated with Nek5000, we have been testing several other configurations ofinterest to NEAMS and NE industry.

4.1 Cold Leg Mixing Benchmark Experiment Configuration.

The Cold Leg Mixing Benchmark experiment resembles a reactor-like geometry where fluid enters the reactorpressure vessel (RPV) during a loss-of-coolant accident. This dangerous scenario of rapidly cooling the vesselis known as pressurized thermal shock (PTS). Evaluation of PTS involves accurately knowing the temperaturedistributions within the cold leg and downcomer regions (Figure 24). The experimental facility simulates thisphenomenon using a closed system, where the only source of energy lies within the potential of the densitydifferences between two isothermal fluids. This means that buoyancy is the only driving force, and the systemwill eventually decay until the concentration homogenizes and velocity reaches zero.

To develop a greater understanding of the flow physics, qualitative observations are also made within thetransfer pipe to describe the complex behavior. The fluid stripe of heavy fluid undergoes interesting behaviorimmediately after the smooth nozzle. Countercurrent flow, instabilities, and the presence of vortex shedding atthe pipe fittings creates at least three transient phases, which can be observed in Figure ?? by instantaneousdensity plots. The Figure ?? (top) shows countercurrent flow that is stable in the wake of the fluid frontbecause of larger concentration gradients due to mixing. The fluid stripe has few oscillations and movesfrom attached-to-outer-wall to detached. In the middle Figure ??, growing Kelvin-Helmholtz-like instabilities

20

Figure 21: Vertical XZ profile of the instantaneous axial velocity flow field at y = 0. Unbalancedflow configuration with larger inlet holes on the left.

21

Figure 22: Horizontal XY profile of the instantaneous axial velocity flow field at H1.

Figure 23: Horizontal XY profile of the instantaneous axial velocity flow field at N1.

22

Figure 24: Isovolume render of (left) heavy fluid and (right) light fluid as they enter both tank andRPV region.

23

Further numerical sensitivity tests using different turbulent diffusion coefficients found that it is necessaryto capture the effects of the molecular Schmidt number between the two different fluids. To stabilize the steepgradients of Sc=1000, the entropy viscosity method (EVM) is implemented into Nek5000, a spectral-elementCFD code. CPU hours were used to run sensitivity tests for the Open Test Case where the density differencesamount to 10%. Furthermore, production runs for the Blind Test Case have been done with 20% densitydifferences at higher resolution for the submission to the NEA/OECD benchmark for CFD verification andvalidation. The results will be published in International Journal of Heat and Fluid Flow.

4.2 Lightbridge Configuration.

Another geometry of interest to NEAMS we have tested here is Lightbridge twisted pin configuration. Asa part of demonstration of our ability to model advanced designs for accident tolerant fuel campaign ofNE industry and NRC, we have considered the TH LES tests of four-loubed pin helical-twist design ofLightbridge.28

Figure 26: Geometry of Lightbridge four-loubed pin with helical twist design.

Figures 27–28 show preliminary LES results volume rendering of instantaneous and time-averaged axialvelocity, correspondingly, in a single twisting pitch triply periodic geometry, while Figures 29–30 show similartemperature fields. Note the same opacity maps in Figures 27–28 and the same colormaps in Figures 29–30are used for instantaneous and time-averaged fields.

Future work will include coupling to MOOSE’s Bison module for fuel-pin conjugate heat transfer solution.

5 Summary

The main focus of this document is to report on Nek5000 code verification and validation (V&V) for twobenchmarks peformed during this and previous year. As a TH component of the SHARP multiphysics toolkit,

28https://ltbridge.com

24

Figure 27: Volume redering of Lightbridge four-loubed pin with helical twist design.

the Nek5000 V&V-driven development aims at assessing the toolkit’s capability of addressing the high-fidelitymodeling and simulation needs identified as important to both the DOE-NE ART program and the nuclearindustry.

For the first benchmark, we have performed the 37-pin PLANDTL sodium experiment LES campaignas part of the advanced M&S activity of CNWG under the U.S.-Japan Bilateral Commission Civil NuclearCooperation on advanced reactor R&D between DOE and JAEA. Run with the latest code release (version17) on up to 1 million MPI ranks, the conjugate heat transfer simulation in the wire-wrap conjugate heattransfer geometry are the largest up-to-date not only with Nek5000 but with any CFD solver. During thesimulation preparation, we have also check the results sensitivity to the outlet boundary condition treatmentand pin thermal conductivity values. Once our JAEA colleagues share the promised experiment date, we willvalidate the simulation results and this code setup for a reference simulation, uncertainty estimation, and forbenchmarking of lower-fidelity faster-turn-around modeling tools.

For the second benchmark, the SESAME INERI 61-pin cross-verification exercise, we extracted anddelivered Nek5000 LES data to our European Horizon 2020’s colleagues from NRG, SCK-CEN, UGent, andENEA. Originally funded by NIC Cost-Share Award, the 61-pin FOA project was a joint venture with AREVA,TerraPower, and TAMU that was aimed at design and completion of well-defined first-of-a-kind validationcampaign for the TH CFD modeling of hexagonal packed fuel assemblies with helically wire-wrapped pin

25

Figure 28: Isovolume render of (left) heavy fluid and (right) light fluid as they enter both tank andRPV region.

geometries typical of SFRs. The LES results will be used by the SESAME INERI partners for RANS-basedmodel comparison and will provide rare transient integral system response from SFR data involving thePhenix Dissymmetric Test, performed by the French CEA with measurements of the reactor systems transientduring a controlled loss-of-coolant from one coolant loop.

Acknowledgments

The work was completed as part of the SHARP reactor performance and safety code suite development project,which is funded under the auspices of the Nuclear Energy Advanced Modeling and Simulation (NEAMS)program of the U.S. Department of Energy (DOE) Office of Nuclear Energy. The work was supported in partby the U.S. DOE under contract DE-NE0008321 and by the U.S. DOE Office of Science, Advanced ScientificComputing Research, under Contract DE-AC02-06CH11357.

These comparisons would not be possible without the Argonne Leadership Computing Challenge (ALCC)program supplying the required computational resources at the Argonne Leadership Computing Facility(ALCF). Some of the computational time support for debugging at large scale of the ALCF capability queuescame from the Exascale Computing Project (ECP) CPU allocation program. We also acknowledge the

26

Figure 29: Isovolume render of (left) heavy fluid and (right) light fluid as they enter both tank andRPV region.

assistance of Yiqi Yu on meshing.

We thank Gail Pieper for editing the manuscript.

27

Figure 30: Isovolume render of (left) heavy fluid and (right) light fluid as they enter both tank andRPV region.

28

References

[1] Haomin Yuan, Jerome Solberg, Elia Merzari, Adam Kraus, and Iulian Grindeanu. Flow-induced vibrationanalysis of a helical coil steam generator experiment using large eddy simulation. Nuclear Engineeringand Design, 322:547–562, 2017.

[2] E. Merzari, H. Yuan, A. Kraus, J. Solberg, and R. M. Ferencz. Coupled calculations in helical steamgenerator: Validation on legacy data. Technical Report ANL/NE-16/49, Argonne National Laboratory,Lemont, IL (United States), 2016.

[3] P. Fischer, J. Lottes, W.D. Pointer, and A. Siegel. Petascale algorithms for reactor hydrodynamics. J.Phys. Conf. Series, 125(1), 2008.

[4] M.O. Deville, P.F. Fischer, and E.H. Mund. High-Order Methods for Incompressible Fluid Flow.Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2002.

[5] Paul F Fischer. An overlapping schwarz method for spectral element solution of the incompressibleNavier-Stokes equations. Journal of Computational Physics, 133(1):84 – 101, 1997.

[6] Paul F. Fischer and Anthony T. Patera. Parallel spectral element solution of the Stokes problem. Journalof Computational Physics, 92(2):380–421, 1991.

[7] Y. Maday and A. T. Patera. Spectral element methods for the incompressible Navier-Stokes equations.In State-of-the-Art Surveys on Computational Mechanics, pages 71–143, 1989.

[8] A. G. Tomboulides, J. C. Y. Lee, and S. A. Orszag. Numerical simulation of low Mach number reactiveflows. Journal of Scientific Computing, 12(2):139–167, June 1997.

[9] Ananias G. Tomboulides and Steven A. Orzag. A quasi-two-dimensional benchmark problem for lowMach number compressible codes. Journal of Computational Physics, 146(2):691–706, 1998.

[10] S. Stolz, P. Schlatter, and L. Kleiser. High-pass filtered eddy-viscosity models for large-eddy simulationsof transitional and turbulent flow. Physics of Fluids, 17(6), 2005.

[11] J. Walker, E. Merzari, A. Obabko, P. Fischer, and A. Siegel. Accurate prediction of the wall shear stressin rod bundles with the spectral element method at high Reynolds numbers. International Journal ofHeat and Fluid Flow, 50:287–299, 2014.

[12] E. Merzari, P. Fischer, and W.D. Pointer. Numerical simulation and proper orthogonal decompositionof the flow in a counterflow T–junction. Journal of Fluid Engineering, 135:091304, 2013.

[13] O. Pironneau, J. Liou, and T. Tezduyar. Characteristic-Galerkin and Galerkin/least-squares space-timeformulations for the advection-diffusion equation with time-dependent domains. Computer Methods inApplied Mechanics and Engineering, 100(1):117–141, 1992.

29

5

Mathematics and Computer Science DivisionArgonne National Laboratory9700 South Cass Avenue, Bldg. 240Lemont, IL 60439

www.anl.gov