Reproducible and replicable CFD: it’s harder than you think

Olivier Mesnard, Lorena A. BarbaMechanical and Aerospace Engineering, George Washington University, Washington DC 20052

Completing a full replication study of our previously published findings on bluff-body aerodynamicswas harder than we thought. Despite the fact that we have good reproducible-research practices,sharing our code and data openly. Here’s what we learned from three years, four CFD codes andhundreds of runs.

Our research group prides itself for hav-ing adopted Reproducible Researchpractices. Barba (2012)1 made a pub-lic pledge titled “Reproducibility PI

Manifesto” (PI: Principal Investigator), which atthe core is a promise to make all research mate-rials and methods open access and discoverable:releasing code, data and analysis/visualizationscripts.

In 2014, we published a study on Physics ofFluids titled “Lift and wakes of flying snakes”.2 Itis a study that used our in-house code for solv-ing the equations of fluid motion in two dimen-sions (2D), with a solution approach called the“immersed boundary method.” The key of sucha method for solving the equations is that it ex-changes complexity in the mesh generation stepfor complexity in the application of boundaryconditions. It makes it possible to use a sim-ple mesh for discretization (structured Cartesian),but at the cost of an elaborate process that in-terpolates values of fluid velocity at the bound-ary points to ensure the no-slip boundary condi-tion (that fluid sticks to a wall). The main find-ing of our study on wakes of flying snakes wasthat the 2D section with anatomically correct ge-ometry for the snake’s body experiences lift en-hancement at a given angle of attack. A previ-ous experimental study3 had already shown thatthe lift coefficient of a snake cross section in awind tunnel gets an extra oomph of lift at 35 de-grees angle-of-attack. Our simulations showedthe same feature in the plot of lift coefficient.4

Many detailed observations of the wake (visual-ized from the fluid-flow solution in terms of thevorticity field in space and time) allowed us togive an explanation of the mechanism providing

extra lift. It arises from a vortex on the dorsalside of the body remaining closer to the surfaceunder the effects of interactions with secondaryvorticity. The flow around the snake’s body crosssection adopts a pattern known as a von Karmanvortex street. It is a particularly complex flow, be-cause it involves three shear layers: the bound-ary layer, a separating free shear layer, and thewake.5 Physically, each of these shear layers issubject to instabilities. The free shear layer can ex-perience 2D Kelvin-Helmholtz instability, whilethe wake experiences both 2D and 3D instabilitiesand can show chaotic behavior. Such flows areparticularly challenging for computational fluiddynamics (CFD).

When a computational research group pro-duces this kind of study with an in-house code,it can take one, two or even three years to write afull research software from scratch, and completeverification and validation. Often, one gets thequestion: why not use a commercial CFD pack-age? Why not use another research group’s open-source code? Doesn’t it take much longer to writeyet another CFD solver than to use existing code?Beyond reasons that have to do with inventingnew methods, it’s a good question. To exploreusing an existing CFD solver for future research,we decided to first complete a full replication ofour previous results with these alternatives. Ourcommitment to open-source software for researchis unwavering, which rules out commercial pack-ages. Perhaps the most well known open-sourcefluid-flow software is OpenFOAM, so we set outto replicate our published results with this code.A more specialist open-source code is IBAMR, aproject born at New York University that has con-tinued development for a decade. And finally,

1

arX

iv:1

605.

0433

9v3

[ph

ysic

s.co

mp-

ph]

15

Oct

201

6

our own group developed a new code, imple-menting the same solution method we had be-fore, but providing parallel computing via therenowned PETSc library. We embarked on a fullreplication study of our previous work, usingthree new fluid-flow codes.

This is the story of what happened next: threeyears of dedicated work that encountered adozen ways that things can go wrong, conqueredone after another, to arrive finally at (approxi-mately) the same findings and a whole new un-derstanding of what it means to do “reproducibleresearch” in computational fluid dynamics.

Story 1: Meshing and boundary con-ditions can ruin everything

Generating good meshes for discretization isprobably the most vexing chore of computationalfluid dynamics. And stipulating boundary con-ditions on the edge of a mesh takes some nerve,too. An early example of how frustrating it can beto investigate different outflow boundary condi-tions is reported in Sani et al. (1994).6 Our first at-tempts at a full replication study of the 2D snakeaerodynamics with IcoFOAM, the incompress-ible laminar Navier-Stokes solver of OpenFOAM,showed us just how vexing and unnerving theseissues can be.

OpenFOAM can take various types of meshas input. One popular mesh generator is calledGMSH: it produces triangles that are as fine asyou want them near the body, while gettingcoarser as the mesh points are farther away. Al-ready, we encounter a problem: how to create amesh of triangles that gives a comparable reso-lution to that obtained with our original struc-tured Cartesian mesh? After dedicated effort, weproduced the best mesh we could that matchesour previous study in the finest cell width nearthe body. But when using this mesh to solvethe fluid flow around the snake geometry, wegot spurious specks of high vorticity in placeswhere there shouldn’t be any (Figure 1). Eventhough the meshes passed the OpenFOAM qual-ity checks, these unphysical vortices appeared forany flow Reynolds number or body angle of at-tack we tried—although they were not respon-sible for the simulations to blow up (fail due torapid error growth). Finally, we gave up with the(popular) GMSH and tried another mesh gener-ator: SnappyHexMesh (details and plots of themeshes are included in the supplementary ma-

terials). Success! No unphysical patches in thevorticity field this time. But another problem per-sisted: after the wake vortices hit the edge of thecomputational domain in the downstream side, anasty back pressure appeared there and startedpropagating to the inside of the domain (Figure2). This situation is also unphysical, and we werecertain there was a problem with the chosen out-flow boundary condition in OpenFOAM, but didnot find any way to stipulate another, more ap-propriate boundary condition. We used a zero-gradient condition for the pressure at the out-let (and tried several other possibilities), whichwe found was a widespread choice in the ex-amples and documentation of OpenFOAM. Af-ter months, one typing mistake when launchinga run from the command line made OpenFOAMprint out the set of available boundary condi-tions, and we found that an advective conditionwas available that could solve our problem. Allthis time, we were looking for a convective con-dition, which is just another name for the samething: satisfying a linear convection equation atthe boundary points. Finally, simulations withOpenFOAM were looking correct—and happily,the main feature of the aerodynamics was repli-cated: an enhanced lift coefficient at 35 degreesangle-of-attack (Figure 3). But not all is perfect.The time signatures of lift and drag coefficientdo show differences between our IcoFOAM cal-culation and the original published ones (Figure4). The key finding uses an average lift coefficient,calculated with data in a time range that is rea-sonable but arbitrary. Refining the mesh or re-ducing the exit criterion of the iterative solversmade a difference of less than 0.5% in this quan-tity. The average force coefficients match (within< 3%) our previous results, despite the differencesseen on the time series. Are these the same solu-tions? Is it acceptable as a replication study? Wethink yes, but this is a judgement call.

Postmortem. IcoFOAM solves the fluid equa-tions using a finite-volume method in an un-structured grid, while our published study usedan immersed boundary method in a stretchedCartesian grid. Comparing results obtained un-der such different conditions is a delicate opera-tion. We made our best attempt at creating a fluidmesh for OpenFOAM that was of similar resolu-tion near the body as we had used before. But un-structured grids are complex geometrical objects.Two unstructured meshes built with the same pa-rameters will not be exactly the same, even. The

2

Fluid-flow solvers we used:

cuIBM— Used for our original study (Krishan et al., 2014), this code is written in C CUDA to exploit GPU hardware, butis serial on CPU. It uses the NVIDIA Cusp library for solving sparse linear systems on GPU. https://github.com/barbagroup/cuIBM

OpenFOAM— A free and open-source CFD package that includes a suite of numerical solvers. The core discretizationscheme is a finite-volume method applied on mesh cells of arbitrary shape. http://www.openfoam.orgIBAMR— A parallel code using the immersed boundary method on Cartesian meshes, with adaptive mesh refinement.https://github.com/ibamr/ibamr

PetIBM— Our own re-implementation of cuIBM, but for distributed-memory parallel systems. It uses the PETSc library forsolving sparse linear systems in parallel. https://github.com/barbagroup/PetIBM

Figure 1: Vorticity field after 52 time-units of flow-simulationwith IcoFOAM for a snake’s section with angle-of-attack 35degrees and Reynolds number 2000. We created a triangu-lar mesh (about 700k triangles) with the free software GMSH.The box insert at the bottom-right shows a zoom-in to the por-tion of the mesh with spurious vorticity (seen bottom-center ofthe main plot).

mesh-generation procedures are not necessarilydeterministic, and regularly produce bad trian-gles that need to be repaired. The complicationsof building a good quality mesh is one of the rea-sons some prefer immersed boundary methods!

Story 2: You can hit snags with otherresearchers’ codes

Open-source research software can often bepoorly documented and unsupported, and on oc-casion it can even be an unreadable mess. But inthis case, we are in luck. IBAMR is a solid pieceof software, the code is documented, and you caneven get swift response from the authors via thetopical online forum. The developers don’t pro-vide a user’s manual, but they have plenty of ex-amples within the code repository. Still, master-ing other researchers’ code is challenging and we

Figure 2: Pressure field after 52 (top) and 53 (bottom) time-units of flow-simulation with IcoFOAM for snake section withangle-of-attack 35 degrees and Reynolds number 2000. Thesimulation crashed after about 62 time-units because of theback pressure at the outlet boundary.

hit a couple of snags that complicated the journey.IBAMR is described as “an adaptive and

distributed-memory parallel implementation ofthe immersed boundary method.” The essenceof the immersed boundary method is that thefluid is represented by a structured mesh, whilethe solid boundary is represented by its own,separate mesh that moves with the body. Wespeak of an Eulerian mesh for the fluid, and a La-grangian mesh for the solid. The forces exertedby the fluid on the body, and vice versa, appearas an additional integral equation and interpola-tion schemes between the two meshes. The roleof these is to make the fluid “stick” to the wall(no-slip boundary condition) and allow the bodyto feel aerodynamic forces (lift and drag). OurcuIBM code uses a variant called the immersed-boundary projection method.7 IBAMR is a librarythat provides different methods,8 but despite thevariations, we assumed it would work similarly.

3

0.8

1.0

1.2

1.4

1.6

1.8

dra

g c

oeff

icie

nt

25o 30o 35o 40o

angle of attack

1.4

1.5

1.6

1.7

1.8

1.9

2.0

2.1

2.2

lift

coeff

icie

nt

Re = 1000

IcoFOAM

Krishnan et al. (2014)

Re = 2000

IcoFOAM

Krishnan et al. (2014)

Figure 3: Time-averaged drag (top) and lift (bottom) coeffi-cients as function of the snake’s angle-of-attack for Reynoldsnumbers 1000 and 2000. We averaged all IcoFOAM force co-efficients between 32 and 64 time-units of flow-simulation aswe have done in our previous study.

We already know that boundary conditionsat the outlet of the computational domain canbe problematic. This is no different with im-mersed boundary methods. Our first attemptwith IBAMR used a boundary condition at theoutlet following their example for flow around acircular cylinder (this turned out to be a traction-free boundary condition). Unfortunately, it re-sulted in a spurious blockage of the wake vorticeswhen they reach the domain boundary: strongvorticity rebounded from the artificial boundaryand propagated back to the domain (Figure 5,top). Of course, this is unphysical and the resultis unacceptable.

In a conversation with the main developers onthe online forum, they suggested a work-around:using “boundary stabilization,” which adds aforcing to push the vortices out. (IBAMR doesnot yet provide a convective/advective boundarycondition.) With this new configuration, the sim-ulations of the snake profile resulted in a wakethat looked physical (Figure 5, bottom), but acomputed lift coefficient that was considerablydifferent from our published study (Figure 6).Another dive into Bhalla et al. (2015) led us tonotice that the benchmark examples were set up

Figure 4: Instantaneous force coefficients on the snake’s sec-tion with angle-of-attack 30 degrees (top) and 35 degrees(bottom) at Reynolds number 2000. We compare the Ico-FOAM results with the cuIBM results from our previous study.We created a 3.4 million cells (mostly hexahedra) with Snap-pyHexMesh, one of the OpenFOAM mesh utilities.

in a way unexpected to us: the no-slip condi-tion is forced inside the body, and not just onthe boundary. Immersed boundary methods nor-mally apply the no-slip constraint on boundarypoints only. When we followed their examples,our simulations with IBAMR were able to repli-cate the lift enhancement at 35 degrees angle-of-attack, although with a slightly different value ofaverage lift (< 5% off). If we look at the timesignature of the lift and drag coefficients, thereis excellent agreement with our previous resultsfor 30 degrees angle-of-attack (Re=2000). But at35 degrees, the time signatures drift apart afterabout 40 time units (more than 150 thousand timesteps). There is a marked drop in the (time vary-ing) lift coefficient (Figure 7), but because the av-erage is calculated over a time range between32 and 64 time units (a reasonable but arbitrarychoice), the final numeric result is not far off ourpublished study. To start, we matched the meshresolution in the vicinity of the body. Refining themesh further, reducing the exit criterion of the it-erative solver, or enlarging the computational do-main did not improve things. Reducing the time

4

Figure 5: Vorticity field after about 61 time-units of flow-simulation with IBAMR for a snake’s section with angle-of-attack 35 degrees and Reynolds number 2000. Top: withoutboundary stabilization at the outlet; bottom: with boundarystabilization.

increment, however, did. In Figure 7, we showthe time-varying lift coefficients obtained withthe parameter CFL set to 0.3 and 0.1—the CFL, orCourant-Friedrichs-Lewy number, constrains theratio of time increment to grid spacing. Like inthe previous case, using OpenFOAM, we make ajudgement call that our result with IBAMR doesindeed pass muster as a replication of our previ-ous study.

Postmortem. Even the best open-source re-search code can have unexpected attributes thatonly the original authors know in depth. We stilldon’t understand why IBAMR requires interiorbody points to be constrained, despite insistentreading of the literature. One issue that affectsour community is that we don’t expect authors toprovide in their papers all the details, nor do werequire papers to be accompanied by code anddata. We learned from this experience that us-ing an open research code and getting correct re-sults with it could involve a long investigative pe-riod, potentially requiring communication withthe original authors and many failed attempts. Ifthe code is not documented and the original au-thors not responsive to questions, then buildingyour own code from scratch could be more sensi-ble!

0.8

1.0

1.2

1.4

1.6

1.8

dra

g c

oeff

icie

nt

25o 30o 35o 40o

angle of attack

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

lift

coeff

icie

nt

Re = 1000

surface markers

body markers

Krishnan et al. (2014)

Re = 2000

surface markers

body markers

Krishnan et al. (2014)

body markers (CFL=0.1)

Figure 6: Time-averaged drag (top) and lift (bottom) coeffi-cients as function of the angle-of-attack of the snake’s sectionfor Reynolds numbers 1000 and 2000. We averaged eachforce signal between 32 and 64 time-units of flow-simulationwith IBAMR to compare with our previous results.

Figure 7: Instantaneous force coefficients at Reynolds num-ber 2000 for the snake’s section at angle-of-attack 30 (top) and35 (bottom) degrees. Here, the no-slip condition is enforcedinside the section. We compare the IBAMR results with cuIBMones from our past study.

5

Story 3: All linear algebra libraries arenot created equal

Our previous study used cuIBM, running on asingle GPU device. The largest problem that wecan fit in the memory of a high-end GPU has justa few million mesh points, which is not enoughto solve three-dimensional flows. We developedPetIBM, a code that uses the same mathemati-cal formulation as cuIBM, to allow solving largerproblems on distributed CPU systems. SincePetIBM and cuIBM implement exactly the samenumerical method, you’d expect that giving thetwo codes the same mesh with the same initialconditions will result in the same solution (withinfloating-point error). Not so fast! We rely on ex-ternal libraries to solve sparse linear systems ofequations: Cusp for GPU devices and PETSc fordistributed CPU systems. It turns out, the itera-tive solvers may have differences that affect thefinal solution.

When repeating our previous simulations ofthe aerodynamics of a snake cross-section withPetIBM, the solutions do not always match thosecomputed with cuIBM. At a Reynolds number of1000, both the time-averaged lift and drag coeffi-cients match. But at Reynolds equal to 2000, av-erage lift and drag match up to 30 degrees angle-of-attack, but not at 35 degrees. That means thatwe don’t see lift enhancement (Figure 8) and themain finding of our previous study is not fullyreplicated. Looking at the time evolution of theforce coefficients for the simulation with PetIBMat Re=2000 and 35 degrees angle-of-attack, wesee a marked drop in lift after 35 time units (topgraph in Figure 9). What is different in the twocodes? Apart from using different linear algebralibraries, they run on different hardware. Leavinghardware aside for now, let’s focus on the itera-tive solvers. Both Cusp and PETSc use the sameconvergence criterion. This is not always the case,and needs to be checked! We’re also not usingthe same iterative solver with each library. ThecuIBM runs (with Cusp) used an algebraic multi-grid preconditioner and conjugate gradient (CG)solver for the modified-Poisson equation. WithPETSc, the CG solver crashed because of an in-definite preconditioner (having both positive andnegative eigenvalues), and we had to select a dif-ferent method: we used a bi-CG stabilized al-gorithm (while still using an algebraic multigridpreconditioner).

Could this difference in linear solvers affect our

unsteady fluid-flow solution? The solutions withboth codes match at lower angles of attack (andlower Reynolds numbers), so what is going on?Because PetIBM and cuIBM use the same method,we don’t need to repeat mesh-convergence anal-ysis. But we did confirm convergence of the so-lution with respect to the exit criterion of the it-erative solvers. We checked everything multipletimes. In the process, we did find some small dis-crepancies. Even a small bug (or two). We found,for example, that the first set of runs with PetIBMcreated a slightly different problem set-up, com-pared with our previous study, where the bodywas shifted by less than one grid-cell width. Ro-tating the body to achieve different angles of at-tack was made around a different center, in eachcase (one used the grid origin at 0,0 while theother used the body center of mass). This tiny dif-ference does result in a different average lift coef-ficient (bottom graph in Figure 9)! The time sig-nal of lift coefficient shows that the drop we wereseeing at around 35 time units now occurs closerto 50 time units, resulting in a different value forthe average taken in a range between 32 and 64.Again, this range for computing the average is achoice we made. It covers about ten vortex shed-ding cycles, which seems enough to calculate theaverage if the flow is periodic. What is caus-ing the drop in lift? Visualizations of the wakevortices (Figure 10) show that a vortex-mergingevent occurs in the middle of the wake, changingthe near-wake pattern. The previously alignedpositive and negative vortices are replaced by awider wake with a single clockwise vortex on thetop side and a vortex dipole on the bottom part.With the change in wake pattern comes a drop inthe lift force.

Postmortem. Although PetIBM implementsthe same immersed-boundary method and wasdeveloped by the same research group, we werenot able to fully replicate the previous findings.The aerodynamic lift on a snake section at 35degrees angle-of-attack is a consequence of thenear-wake vortices providing extra suction on theupper side of the body. When a vortex mergerevent changes the wake pattern, lift drops. Vor-tex merging is a fundamentally two-dimensionalinstability, so we expect that this problem won’ttrouble us in more realistic 3D simulations. Butit is surprising that small changes—within thebounds of truncation error, roundoff error and al-gebraic errors—can trigger this instability, chang-ing the flow appreciably. Even when the only

6

0.8

1.0

1.2

1.4

1.6

1.8

dra

g c

oeff

icie

nt

25o 30o 35o 40o

angle of attack

1.4

1.6

1.8

2.0

2.2

lift

coeff

icie

nt

Re = 1000

displaced markers

Krishnan et al. (2014)

Re = 2000

displaced markers

Krishnan et al. (2014)

original markers

Figure 8: Time-averaged drag (top) and lift (bottom) co-efficients as function of the snake’s angle-of-attack and forReynolds numbers 1000 and 2000 using the same Eule-rian mesh as in our past cuIBM simulations. We showPetIBM results obtained when the immersed-boundary is ro-tated around: (1) its center of mass (green and orange sym-bols) and (2) the reference origin (solo red marker).

difference between two equivalent simulations isthe linear algebra library used, there can be chal-lenges to reproducibility.

Story 4: Different versions of yourcode, external libraries or even com-pilers may challenge reproducibility

In the span of about three years, we ran more than100 simulations with OpenFOAM, IBAMR, andPetIBM, encountering about a dozen things thatcan go wrong. We replicated our previous scien-tific finding (enhanced lift at 35 degrees angle-of-attack for sufficiently large Reynolds number) intwo out of three campaigns. Ironically, the casethat did not replicate our findings was that of ourown code re-write. The original code (cuIBM)and the re-write (PetIBM) use different linear al-gebra libraries, and it’s unnerving to think thiscould change our results. This final story is aboutwhat happened when we went back to our orig-inal code and tried to reproduce the publishedfindings.

As we mentioned in the opening of this article,

Figure 9: Instantaneous force coefficients for the snake’s sec-tion with angle-of-attack 35 degrees and Reynolds number2000. The top plot compares the PetIBM results with thosereported in our previous study. The bottom plot shows resultswith the immersed-boundary being rotated around the refer-ence origin or around its center of mass (the latter is slightlyshifted compared to our previous study).

we adopted a set of practices years ago to makeour research reproducible. The study publishedas “Lift and wakes of flying snakes” followed theguidance of the “Reproducibility PI Manifesto,”which includes: (1) code developed under ver-sion control; (2) completed validation and veri-fication, with report published on Figshare; (3)open data and figures for the main results of thepaper on Figshare; (4) pre-print made availableon arXiv; (5) code released under MIT License;(6) a Reproducibility statement in the paper. Theoriginal work, of course, confirmed grid indepen-dence of the solution: Krishnan et al. (2014) re-port differences in the average lift coefficients inthe order of 2% at 35 degrees angle-of-attack and< 0.1% at 30 degrees. Naturally, we expected tobe able to reproduce our own results!

The first hurdle we faced is that, three years af-ter we completed our previous study, we haveupdated our lab computers: new operating sys-tems, new GPU devices, new external libraries.The code itself has been modified to implementnew features. Happily, we have version con-

7

Figure 10: Vorticity field after 19, 52, 53, and 64 time-units offlow-simulation with PetIBM for a snake’s section at angle-of-attack 35 degrees and Reynolds number 2000. The vortex-merging event is responsible for the change in the wake sig-nature and the drop in the mean lift coefficient.

trol. So, we set out to reproduce our results withcuIBM using (1) the “same” old version of thecode and (2) the current version. In both cases,we used identical input parameters (Lagrangianmarkers to discretize the geometry, grid parame-ters, flow conditions, and solver parameters). Butthe three-year-old simulations used a version ofCusp (0.3.1) that is no longer compatible with theoldest CUDA version installed on our machines(5.0). Thus, we adapted “old” cuIBM to be com-patible with the oldest version of Cusp (0.4.0) thatwe can run. The case at angle-of-attack 35 de-grees and Reynolds number 2000 now gave anappreciable difference compared with our pre-vious study: the instantaneous force coefficientsstart to slowly drop after about 60 time units (Fig-ure 11(c)). Now, this is really the same code, with

only a difference in the version of the linear al-gebra library. Repeating the case with the mostcurrent version of cuIBM and the same version ofCusp (0.4.0) leads to the same force signals, with aslight drop towards the end (Figure 11(d)). Andthe same is the case with the current version ofcuIBM and a later version of Cusp (0.5.1). The fi-nal findings in these cases do not vary from ourpublished work: there is, in fact, lift enhancementat 35 degrees angle-of-attack . . . but the resultsmatch only because we calculate the average liftin a time interval between 32 and 64. Yet, the flowsolution was affected by changing the version ofa dependent library. (The revision history of Cuspsays that they refactored the smooth-aggregationsolver between the two versions we are using.)The hardware was also different (a K20 GPU ver-sus a C2070 in our older study), and the operat-ing system, and the compiler. (Note that we al-ways run in double precision.) In an iterative lin-ear solver, any of these things could be related tolack of floating-point reproducibility. And in un-steady fluid dynamics, small floating-point dif-ferences can add up over thousands of time stepsto eventually trigger a flow instability (like vortexmerging).

Postmortem. Making research codes opensource is not enough for reproducibility: we mustbe meticulous in documenting every dependencyand the versions used. Unfortunately, some ofthose dependencies will get stale over time, andmight cease to be available or usable. Your ap-plication code may give the same answer with adifferent version of an external library, or it maynot. In the case of unsteady fluid dynamics, thenonlinear nature of the equations combined withnumerical non-reproducibility of iterative linearsolvers (in parallel!) can change the results.

Lessons learned

Reproducibility and replication of studies are es-sential for the progress of science, and much ofscience today advances via computation. Weuse computer simulations to create new knowl-edge. How can we certify that this new knowl-edge is justified, that there is enough evidenceto support it? The truth is computational sci-ence and engineering lacks an accepted standardof evidence. Some previous efforts in CFD havesought to compare results from multiple codes:e.g., Dimonte et al. (2004)11 report results forRayleigh-Taylor instability using seven different

8

Figure 11: Instantaneous force coefficients on the snake’s section at Reynolds number 1000 and angle 35 degrees (top-left)and Reynolds number 2000 and angle 30 degrees (top-right). Bottom-left: instantaneous force coefficients at Reynolds number2000 and angle-of-attack 35 degrees running the same version of cuIBM (adapted to CUSP-0.4.0) than the one used for ourprevious study. Bottom-right: drop of the mean force coefficients observed over the end of the simulation using two versions ofcuIBM (“current” and “old”) with different CUSP versions (0.4.0 and 0.5.1).

Definition of reproducible research:

The literature is rife with confused and sometimes contradictory meanings for reproducible research, reproducibility, replicability,repetition, etc. It is thus worth clarifying what we mean by these terms. The phrase “reproducible research” in reference tocomputational studies that can be reproduced by other scientists was introduced by geophysics professor Jon Claerbout in the1990s. An early volume of CiSE published an article about the reproducible-research environment they created in his group atStanford. 9 Their goal was complete documentation of scientific computations, in such a way that a reader can reproduce all theresults and figures in a paper using the author-provided computer programs and raw data. This ability requires open data andopen-source software, and for this reason the reproducibility movement is closely linked with the open science movement. Infollowing years, the term replication was distinctly adopted to refer to an independent study, re-running experiments to generatenew data that, when analyzed, leads to the same findings. We follow this convention, clarified more recently in an article inScience. 10

Reproducible research— Authors of a study provide their code and data, allowing readers to inspect and re-run the analysisto recreate the figures in the paper. Reproducible research makes replication easier.Replication— An independent study that generates new data, using similar or different methods, and analyzes it to arrive atthe same scientific findings as the original study.

CiSE has dedicated two theme issues to reproducibility: January/ February 2009 and July/August 2012.

9

codes (from five institutions), while the AIAAdrag-prediction workshop12 and the high-lift-prediction workshop13 have been collecting re-sults for years using a variety of commercial andnon-commercial (mostly closed) software. How-ever, these efforts don’t have a specific goal ofreplicating a published finding nor are they con-cerned with reproducible workflows. We labelcomputational research reproducible when authorsprovide all the necessary data and the computercode to run the analysis again, re-creating the re-sults. But what data are necessary? We found thatopen-source code and open data sets are a min-imal requirement. Exhaustive documentationduring the process of computational research iskey. This includes documenting all failures. Cur-rent publication custom is biased towards posi-tive results.14 The CFD community does not havea habit of communicating negative results; onerare example is the analysis of Godunov methodsand its failures by Quirk (1997).15 In the case ofIBAMR, negative results with points only on theboundary are not among the examples provided:the situation may be obvious to the authors, butnot to the users. We learned how importantthe computational mesh and the boundary con-ditions can be. A reproducible computational pa-per should include the actual meshes used in thestudy (or a deterministic mesh-generation code)and careful reporting of boundary conditions.This is rarely (if ever!) the case. We learnedthat in addition to developing our code underversion control, we need to carefully record theversions used for all dependencies. In practice,such careful documentation is feasible only witha fully automated workflow: launching simula-tions via running scripts, storing command-linearguments for every run, capturing complete en-vironment settings. Post-processing and visual-ization ideally should also be scripted, avoidingsoftware GUIs for manipulation of images. Newtools have emerged to help reproducible work-flows; for example, Docker containers to capturethe full state of the operating system, applicationsoftware, and dependencies.

We learned that highly unsteady fluid dynam-ics is a particularly tough application for repro-ducibility. The Navier-Stokes equations are non-linear and can exhibit chaotic behavior under cer-tain conditions (e.g., geometry, Reynolds number,external forcing). Some flow situations are sub-ject to instabilities, like vortex merging in two di-mensions and other vortex instabilities in 3D. In

any application that has sufficient complexity, weshould repeat simulations checking how robustthey are to small variations. And report nega-tive results! Understandably, long 3D simulationsthat take huge computational resources may notbe feasible to repeat. We should continue the con-versation about what it means to do reproducibleresearch in high-performance computing (HPC)scenarios. When large simulations run on specifichardware with one-off compute allocations, theyare unlikely to be reproduced. In this case, it iseven more important that researchers advance to-wards these HPC applications on a solid progres-sion of fully reproducible research at the smallerscales.

Computational science and engineering makesubiquitous use of linear algebra libraries likePETSc, Hypre, Trilinos and many others. Rarelydo we consider that using different librariesmight produce different results. But that is thecase. Sparse iterative solvers use various defini-tions of the tolerance criterion to exit the iterations,for example. The very definition of residual couldbe different. This means that even when we setthe same value of the tolerance, different librariesmay declare convergence differently! This posesa challenge to reproducibility, even if the applica-tion is not sensitive to algebraic error. The situa-tion is aggravated by parallel execution. Globaloperations on distributed vectors and matricesare subject to rounding errors that can accumu-late to introduce uncertainty in the results.

We are recommending more rigorous stan-dards of evidence for computational science andengineering, but the reality is that most CFD pa-pers are not even accompanied by a release ofcode and data. The reasons for this are varied:historical, commercial interests, academic incen-tives, time efficiency, export controls, etc. The ori-gins of CFD in the Los Alamos Laboratory in the1940s was secret research, and when computercode was stored in large boxes of punched cardsor big rolls of magnetic tape, there was hardlya way to “share” it.16 The 1970s saw the birthof commercial CFD, when university professorsand their students founded companies fundedunder the US government’s SBIR program. It’snot unreasonable to speculate that the potentialfor commercial exploitation was a deterrent foropen-source release of CFD codes for a long time.It is only in the last 15 years or so that open-source CFD codes have become available. Butthe CFD literature became entrenched in the habit

10

of publishing results without making availablethe code that generated those results. And now,we face the clash between the academic incentivesystem and the fact that reproducible researchtakes a substantial amount of time and effort.This campaign to replicate our previous resultstaught us many lessons on how to improve ourreproducibility practices, and we are committedto maintaining this high standard. We will con-tinue to share our experiences.

Supplementary materials

We provide supplementary materials in theGitHub repository for this paper, including: (1)all input files to generate the runs reported inthe paper; (2) Jupyter notebooks for all the simu-lations, detailing every condition (dependencies,compilation, mesh, boundary conditions, solverparameters, command-line options); (3) Pythoncodes needed to recreate all the figures includedin the paper. In addition, we separately reportour efforts to assess independence of the solutionwith respect to grid spacing, time increment anditerative tolerance (with each code).

Our codes are available for unrestricted use,under the MIT license; to obtain the codes andrun the tests in this paper, the reader mayfollow instructions on https://github.com/barbagroup/snake-repro.

Acknowledgements

We’re grateful for the support from the US National Sci-ence Foundation for grant number NSF OCI-0946441and from NVIDIA Corp. for equipment donations un-der the CUDA Fellows Program. LAB would like to ac-knowledge the hospitality of the Berkeley Institute ofData Science (BIDS), where this paper was written.

References

[1] Lorena A. Barba. Reproducibility PI Manifesto, 2012.Figshare doi:10.6084/m9.figshare.104539.

[2] Anush Krishnan, John J. Socha, Pavlos P. Vlachos, andL. A. Barba. Lift and wakes of flying snakes. Physics ofFluids, 26(3):031901, mar 2014.

[3] D. Holden, J. J. Socha, N. D. Cardwell, and P. P. Vla-chos. Aerodynamics of the flying snake Chrysopeleaparadisi: how a bluff body cross-sectional shape con-tributes to gliding performance. Journal of ExperimentalBiology, 217(3):382–394, jan 2014.

[4] Anush Krishnan, John J. Socha, Pavlos P. Vlachos, andLorena A. Barba. Lift and drag coefficient versus angleof attack for a flying snake cross-section, 2013. Figsharedoi:10.6084/m9.figshare.705883.v2.

[5] C H K Williamson. Vortex Dynamics in the CylinderWake. Annu. Rev. Fluid Mech., 28(1):477–539, jan 1996.

[6] R. L. Sani and P. M. Gresho. Résumé and remarks on theopen boundary condition minisymposium. InternationalJournal for Numerical Methods in Fluids, 18(10):983–1008, may 1994.

[7] Kunihiko Taira and Tim Colonius. The immersed bound-ary method: A projection approach. Journal of Compu-tational Physics, 225(2):2118–2137, aug 2007.

[8] Amneet Pal Singh Bhalla, Rahul Bale, Boyce E. Grif-fith, and Neelesh A. Patankar. A unified mathemati-cal framework and an adaptive numerical method forfluid–structure interaction with rigid deforming, and elas-tic bodies. Journal of Computational Physics, 250:446–476, oct 2013.

[9] M. Schwab, N. Karrenbach, and J. Claerbout. Makingscientific computations reproducible. Comput. Sci. Eng.,2(6):61–67, 2000.

[10] R. D. Peng. Reproducible Research in ComputationalScience. Science, 334(6060):1226–1227, dec 2011.

[11] Guy Dimonte, D. L. Youngs, A. Dimits, S. Weber, M. Mari-nak, S. Wunsch, C. Garasi, A. Robinson, M. J. An-drews, P. Ramaprabhu, A. C. Calder, B. Fryxell, J. Biello,L. Dursi, P. MacNeice, K. Olson, P. Ricker, R. Rosner,F. Timmes, H. Tufo, Y.-N. Young, and M. Zingale. A com-parative study of the turbulent Rayleigh–Taylor instabilityusing high-resolution three-dimensional numerical simu-lations: The Alpha-Group collaboration. Physics of Flu-ids, 16(5):1668, 2004.

[12] David W. Levy, Kelly R. Laflin, Edward N. Tinoco, John C.Vassberg, Mori Mani, Ben Rider, Christopher L. Rumsey,Richard A. Wahls, Joseph H. Morrison, Olaf P. Broder-sen, Simone Crippa, Dimitri J. Mavriplis, and MitsuhiroMurayama. Summary of Data from the Fifth Computa-tional Fluid Dynamics Drag Prediction Workshop. Jour-nal of Aircraft, 51(4):1194–1213, jul 2014.

[13] Christopher L. Rumsey and Jeffrey P. Slotnick. Overviewand Summary of the Second AIAA High-Lift PredictionWorkshop. Journal of Aircraft, 52(4):1006–1025, jul2015.

[14] John P. A. Ioannidis. Why Most Published ResearchFindings Are False. PLoS Med, 2(8):e124, aug 2005.

[15] James J. Quirk. A Contribution to the Great Rie-mann Solver Debate. In Upwind and High-ResolutionSchemes, pages 550–569. Springer Science BusinessMedia, 1997.

[16] N. Metropolis and E.C. Nelson. Early Computing at LosAlamos. IEEE Annals Hist. Comput., 4(4):348–357, oct1982.

Olivier Mesnard is a doctoral student at the GeorgeWashington University. He has an Engineering degree

11

from Ecole Supérieur de Mécanique (SUPMECA) inParis, and a Master’s degree in Sciences from Uni-versité Pierre et Marie Curie, also in Paris. His re-search interests include computational fluid dynamicsand immersed-boundary methods with application toanimal locomotion.

Lorena A. Barba is Associate Professor of Mechani-cal and Aerospace Engineering at the George Wash-ington University. She obtained her PhD in Aeronau-tics from the California Institute of Technology. Her re-search interests include computational fluid dynamics,especially particle methods for fluid simulation and im-mersed boundary methods; fundamental and appliedaspects of fluid dynamics, especially flows dominatedby vorticity dynamics; the fast multipole method andapplications; and scientific computing on GPU archi-tecture. She received the Amelia Earhart Fellowship, aFirst Grant award from the UK Engineering and Phys-ical Sciences (EPSRC), the National Science Founda-tion Early CAREER award, and was named CUDA Fel-low by NVIDIA, Corp.

12