Comparison of Solution Verification Techniques

Photos placed in horizontal position with even amount

of white space between photos

and header

Photos placed in horizontal position

with even amount of white space

between photos and header

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2011-XXXXP

Comparison of Solution Verification TechniquesV. Gregory Weirs, William J. Rider, James R. KammSandia National Laboratories, Albuquerque, NMJuly 24, 2013*Los Alamos National Laboratory after August 5th 2013

SAND 2013-6079C

2

Acknowledgements Idea of solution verification (also called calculation

verification): Pat Roache, “Verification and Validation in Computational Science and Engineering,” 1998.

An excellent book, incorporating more than a decade of V&V theory development advanced by significant DOE ASC program funding: Bill Oberkampf and Chris Roy , “Verification and Validation in Scientific Computing,” 2010. The canon.

Recent work in this talk was developed by Bill Rider, Walt Witkowski, Jim Kamm, and Yakov Ben-Haim.

3

Outline Context and motivation Code Verification – the theory of numerical error Solution Verification – what it is and what is not The Grid Convergence Index (GCI) The Robust Multi-Regression (RMR) approach The Info-Gap Theory

4

More is Being Asked of Modeling and SimulationOur focus is on engineering and scientific simulations that support decisions of consequence Decisions of consequence: As a result of the decision,

significant money will be spent, lives may be placed at risk Simulations provide information that either can not be

obtained or is too expensive to obtain in another way Decision maker will not have deep knowledge of the

application, and might not have a technical background

This context simply reflects the increasing maturity of modeling and simulation

5

Predictive Modeling and Simulation Require Quantified Uncertainties Numerical simulations provide approximate solutions to

models of reality Validation addresses the sufficiency of a model for a specific

application or intended use Verification addresses numerical error and method

correctness, which arises from the approximate solution techniques employed by FEM and other discretization techniques (FDM, FVM, …)

Uncertainty quantification addresses variability in simulation results due to uncertainty in modeling (often accessed via input parameters, or model selection) UQ should include numerical uncertainty computed via verification

Quite often comparisons are done using the “viewgraph” normThese plots are merely notional schematics that illustrate where we want to go.

Exp. DataBest calculation

6Credit: Rider SAND2013-2059C

Solutions are often described as “converged” as a subjective opinionThis is “mesh sensitivity”. This is not solution verification!

Exp. Data ± ErrorBest calculation

FineMedium


Convergence: the numerical error decreases with finer mesh resolution

FineMediumCoarse

A poor man’s method of solution verification:(where mesh doubling is used…) • Equally spaced lines implies zeroth order• Factor of two decrease implies first order• Factor of four decrease implies second order

Solution Verification for


This is solution verification, despite the negative results

With a third resolution convergence can be assessed.

These results are is NOT converged, assuming mesh doubling (0th order).

Convergence: the numerical error decreases with finer mesh resolution


FineMediumCoarse

9

It is absolutely essential that the quantity “Q” be something that can reasonably be expected to converge numerically.

Credit: Rider SAND2013-2059C

These results are converging (~1st order). Distance between the curves is halved with refinement.

In an ideal world you could extrapolate the solution to h=0.With three grids a convergence rate and a converged solution can estimated.

FineMediumCoarseExtrapolated



“Ideal world” means you have high confidencein the estimate of the observed error.

For practical calculations, use the solutions to estimate numerical error


FineMediumCoarseExtrapolated


Error estimates are useful evenwhen the solutions are not converging

FineMediumCoarse


We want to produce justifiable estimates even in “bad” cases!

13

This Talk Is About the Practical Estimation of Numerical ErrorFor real-world problems simulations are often: not smooth – singular, cusps, shocks, material discontinuities multiphysics multiscale not robust - code might crash limited by resources - small number of simulations, especially

for fine meshes not in the asymptotic range difficult to analyze – it might be difficult to quantitatively

relate quantities of interest (QOIs) to the solution

14

The Cook-Cabot Problem Compressible 1D Euler equations Periodic BCs Wave is nearly acoustic Very low Mach number (< 0.05)

Reference: Cook, A. W. and Cabot, W. H., J. Comp. Phys. 195, 2004.

t=0.0e-5t=0.5e-5t=1.0e-5t=1.5e-5t=2.0e-5

densityvelocity

Exact solution Initially smooth Based on Burgers’

equation Exact solution until

shock formation

Dens

ity

Position

Pres

sure

Position

Velo

city

Position

SIE

Position

IC IC

IC

IC

The Sod ProblemRiemann problems are the canonical shock physics IVP, with two constant initial states. The Sod “shock tube” problem is a very mild test, but has a

discontinuous exact solution The Riemann problem for gas dynamics has three canonical

waves: Rarefaction Contact

Shock

Credit: Kamm, Rider, Weirs, Love, SAND2011-7601C

Reference: Sod, G. A., J. Comp. Phys. 27:1-31, 1978.

16

ALEGRA is used to compute solutions Developed by SNL under the DOE ASC program Designed for: ideal and resistive MHD, shock physics, solid

dynamics; lots of material models Applications: Z-pinchs, hypervelocity impact, shaped-charge

jets, high explosives

Today’s results: just hydrodynamics: Lagrange + Remap; has demonstrated second order accuracy

in time and space for smooth solutions, however, for this work first order algorithms were used

Artificial viscosity for shock capturing - can interfere with design order of accuracy

(Passive) tracer particles provide diagnostics; but might not have the same error behavior as the field solution

17

Code Verification

We have theory and analysis that show our numerical method has certain properties if certain assumptions are met

Code verification is the process for testing whether or not an implementation of the method (i.e., the code) delivers these properties Requires exact solutions of the governing equations Used to find bugs Used to learn more about the numerical method

The convergence rate is the most commonly tested property, used as the primary fiducial, and the most important

Truncation error has a power law dependence on the mesh spacing The standard power-law

form for the error is: Characteristic length scale of mesh cell, “∆x”

Computed solution on grid of mesh size h

Exact solution,i.e., h 0

Convergence rateCoefficient

Credit: Kamm, Witkowski, Rider, Trucano, Ben-Haim , SAND2012-7798C

Almost all mesh-based discretizations reduce to this error ansatz under broadly similar assumptions: h is “small enough”: asymptotic regime Uniform mesh refinement Convergence rate depends on the smoothness of the solution

The details do matter, and numerical analysis of a particular method provides these details

Code Verification: Given eh, h, Find A, p

Asymptotic convergence is the fundamental concept• PDEs are discretized in space Δx, time Δt, etc., for resolution

using finite-digit arithmetic.

This is typically where one wishes to run an analysis code.

Domain where round-offerrors start to accumulate

Slope = -1

Slope = -½

Domain where the discretization is not

adequate to solve the continuous equations

0 10-7 10-6 10-5 10-4 10-3 10-2 10-1

10+1

10+0

10-1

10-2

10-3

10-4

…

10-15

10-16

log(Δx)

log

||yco

ntin

uous

–y(Δx)

||

N O T I O N A L

Domain of asymptotic convergence, where the truncation error

dominates

“Stagnation”

“As we refine the grid we hope to get better approximations to the true solution.” Randy LeVeque, Computational Methods for Astrophysical Fluid Flow

Credit: F. Hemez

Credit: F. Hemez (LANL)

20

Cook-Cabot Code VerificationNX h |e|1 rate

5 0.2 23649210 0.1 94830.9 1.318420 0.05 40825 1.215940 0.025 18926.4 1.109160 0.01667 11573.9 1.212980 0.0125 8541.5 1.0561

100 0.01 6682.6 1.0999120 0.00833 5490.32 1.0779150 0.00667 4356.12 1.0370200 0.005 3232.16 1.0374400 0.0025 1592.91 1.0208

1000 0.001 632.11 1.00872500 0.0004 252.496 1.0015

10000 0.0001 65.3684 0.974825000 0.00004 28.4615 0.9074

Line has many more points than the table, especially for low resolution

Relatively smooth (see low resolution points)

L1 norm of density error

21

Sod Code Verification Results

NX = 10, 15, 20, …, 5000 Noisy; common for

discontinuous solutions Point-wise, noisy rates, but

convergence globally

NX h |e|1 rate

10 0.1 0.0237867

25 0.04 0.0210917 0.131

50 0.02 0.0062699 1.750

100 0.01 0.0036430 0.783

125 0.008 0.0045640 -1.010

250 0.004 0.0014890 1.616

500 0.002 0.0009268 0.684

1000 0.001 0.0005485 0.757

5000 0.0002 0.0003148 0.345

L1 norm of density error

Simulations used default settings in ALEGRA; observe p~1 (globally, not point-wise) with minor adjustments

22

Sod Code Verification ResultsNX h R rate C rate S rate

10 0.1

25 0.04 0.406 -0.623

0.340

50 0.02 1.874 1.768 1.301

100 0.01 1.156 0.892 -0.032

125 0.008 -2.508 -1.341

2.580

250 0.004 2.166 1.153 1.322

500 0.002 0.968 0.566 0.510

1000 0.001 0.776 0.758 0.727

5000 0.0002 0.424 0.421 0.102Different waves have different convergence rates*; eventually, the contact error will dominate the error (in theory)

L1 norms of density error

Rarefaction:Contact:

Shock:

*Reference: Banks, Aslam, Rider, J. Comp. Phys. 227 (2008)

Greg has post-conference homework!

23

Solution VerificationInformal definition of solution verification: Numerical error estimation based on a sequence of simulations at different mesh resolutionsKey features: Non-intrusive – apply to existing codes Relatively inexpensive Applies to solution and functionals, under appropriate

assumptionsOther error estimation techniques (not discussed here) Error transport equation Approaches using the dual problem or adjoints A posteriori error estimation

Code Verification vs. Solution Verification

24

Code Verification:• You have an exact

solution, so you compute exact errors

• You are testing your code (implementation, algorithm)

• Hard estimates of convergence properties

• Metrics are defined by numerical analysis

Solution Verification:• You don’t have an exact

solution, you estimate numerical errors

• You test your solution(s)• Soft estimates of

numerical error• Metrics are defined by the

analyst – integrated quantities, point values, functionals of the solution

25

Richardson Extrapolation Oberkampf and Roy (2010), following Roache:

estimated value

observed ratethree equations

three unknownsprefactor

Constant refinement ratio: Non-constant refinement ratios: no closed form solution; solve iteratively for

More than three meshes: overdetermined system

Solution Verification: Given uh, h, Estimate

Estimate numerical uncertaintyusing the Grid Convergence Index The standard power error ansatz gives an

estimate of numerical error, :

For three or more calculations, p is calculated and the uncertainty is estimated with a “safety factor”:

This result is claimed to give a “95% confidence interval.” This claim is based on many years of experience with simulations.

For two calculations, no estimate for is possible — and the estimated uncertainty is greater:

log (h )

log

(u)

hf hm hc

coarse

medium

fine

with

with

Credit: Kamm, Witkowski, Rider, Trucano, Ben-Haim , SAND2012-7798C

27

Sequence(NX)

Convergence Rate

Extrapolated Solution

Uncertainty Estimate

True Error

80, 90, 100, 110 8.71 0.261569 9E-0699, 100, 101,

102, 103 2.53 0.261282 8.51E-4

16, 18, 20, 22 6.34 0.261605 1.35E-5

20, 40, 60, 80 127.24 0.261024 1.66e-3

5, 10, 20, 40 3.08 0.261554 1.53e-5

6, 9, 12, 15 106.62 0.261756 1.63e-3

Exact value =

Cook Cabot results using the GCIPosition of Lagrangian tracer x0 = 0.25 Computed convergence rates suggest

extrapolation is not reliable Sequences: coarse vs. asymptotic regime,

clustered vs. doubling

28

Density at x = 0.25 Convergence rates again suggest

large uncertainties Two uncertainty estimates are lower

than the true error

Sequence(NX)

Convergence Rate



True Error

80, 90, 100, 110 7.86 1.08006 0.006563 0.00217999, 100, 101,

102, 103 75.86 1.07891 0.002904 0.003329

16, 18, 20, 22 12.39 1.04687 0.05642 0.035369

20, 40, 60, 80 139.19 1.04206 0.07089 0.040179

5, 10, 20, 40 1.56 1.08008 0.0147 0.002159

6, 9, 12, 15 19.10 1.02603 0.03416 0.056209

Cook Cabot results using the GCI

Exact value = 1.08223855

29

Exact value:x = 0.732558

velocitydensity

Sod results using the GCIPosition of Lagrangian tracer x0 = 0.60 Computed convergence rates suggest

extrapolation is not reliable One uncertainty estimate is lower than

the true error

Sequence(NX)

Convergence Rate



True Error

20, 40, 80, 160 124.64 0.731883 ±0.00262575 0.0006754

15, 30, 60, 120 12.89 0.726353 ±0.019296 0.0062054

25, 30, 35, 40 10.99 0.731432 ±0.0025245 0.0011264

80, 90, 100, 110 77.62 0.732734 ±0.0000615 -0.0001756

80, 160, 320, 640 0.012 0.72848 ±0.0213774 0.0040784

85, 170, 340, 680 7.85 0.731899 ±0.002112 0.0006594

30


Sequence(NX)

Convergence Rate



True Error

20, 40, 80, 160 5.68 0.489701 ±0.0749882 0.0045748

15, 30, 60, 120 0.79 0.476967 ±0.040598 0.0173088

25, 30, 35, 40 10.91 0.506525 ±0.0538432 -0.0122492

80, 90, 100, 110 8.19 0.498984 ±0.00132525 -0.0047082

80, 160, 320, 640 8.17 0.491581 ±0.00503175 0.0026948

85, 170, 340, 680 7.95 0.496213 ±0.006783 -0.0019372

Density at x = 0.45 Convergence rates again suggest large

uncertainties One uncertainty estimate is lower than

the true error

Sod results using the GCI

Why verification analyses cango so very, very wrong. The data in this study is poorly conditioned: the grids are

too close in resolution, and far from the asymptotic range. Such poor conditioning produces unreliable solutions. The super-high convergence rate is not ruled out, but could

be. The use of constraints on the solution for the convergence rate have the benefit of avoiding such absurd outcomes.

Once constraints are added to the problem, the solution no longer is unique. The norm that is minimized to obtain the solution changes the solution significantly.

Introducing these changes across a set of norms gives a diversity of results that requires a rethink of how such analysis is conducted.

31Credit: Rider, Kamm, Witkowski, Ben-Haim , SAND2013-2059C

Remember, both the data and the regression can produce outliers.

Regression, norms and probability distributions

Minimization of the residual for regression carries implications about optimality. The fit is optimal if the errors are distributed: Gaussian implies L2 , the standard

approach (unweighted) Laplace (double exponential) implies L1

(absolute value) Uniform implies Linfinity (maximum)

Regression can be done in any norm if the data is either under- or over-determined and can include constraints as well.

Usual statistics are highly susceptible to outliers, but median statistics are not.

L1 Regression Laplace PDF

GaussianPDF

Linf RegressionUniform PDF

32

Use robust statistics not standard statistics

Credit: Rider, Kamm, Witkowski, Ben-Haim , SAND2013-2059C

Verification uses (nonlinear) regression to compute everything. Typically, a standard least squares fit to the data is used.

Assumptions are usually not stated, nor necessarily appropriate. We apply the following algorithm to the data (2 or more grids):1. Define the theoretical convergence rate, pth, with lower (pL) and

upper (pU) bounds on the expected convergence rate.2. Solve the regression problem in multiple norms ranging from L1

to Linfinity with the p constrained to be in[pL, pU]. 3. Also solve the linear regression where the convergence rates are

defined as pth, pL and pU.4. From the set of and p select the median values and the median

deviation for each: 5. Solve for the absolute error in the same manner to obtain:

Apply robust statistical techniques to produce results with quantified confidence. Run over subsets of data (jackknife).

Robust multi-regression method: robust statistics plus expert knowledge

A structured, principled way to introduce specific expert knowledge. Bounds on extrapolated solution could be used, too (such as, e.g., positivity).

Produce a set of estimates with explicitly defined assumptions about statistics of the error; this set of estimates is free of the fragility of a single estimate.

Produce a bounding estimate using the same approach. Useful when the data is non-monotonic.

The standard method is fragile and includes implicit assumptions regarding statistics, error and convergence, these assumptions are not included in the analysis.

33Credit: Rider, Kamm, Witkowski, Ben-Haim , SAND2013-2059C

34

Exact value =

Cook Cabot results using RMRPosition of Lagrangian tracer x0 = 0.25

Sequence(NX)

Convergence Rate

Rate Uncertainty


Solution Uncertainty

True Error

80, 90, 100, 110 2.77 0.56 0.261576 8.5e-699, 100, 101,

102, 103 2.33 1.80 0.261576 4.49e-4

16, 18, 20, 22 2.53 0.14 0.261564 9.45e-4

20, 40, 60, 80 2.22 1.06 0.261602 2.80e-4

5, 10, 20, 40 2.77 0.59 0.261549 3.11e-5

6, 9, 12, 15 2.29 1.69 0.261575 1.28e-4

Rates more reasonable Uncertainty estimates smaller than those

for GCI RMR provides uncertainty estimate for

the rate

35


Cook Cabot results using RMR

Sequence(NX)

Convergence Rate

Rate Uncertainty



True Error

80, 90, 100, 110 2.76 0.61 1.08317 0.01180 -0.00093199, 100, 101,

102, 103 2.91 0.28 1.07991 0.05194 0.002328

16, 18, 20, 22 2.24 0.92 1.09151 0.07515 -0.009271

20, 40, 60, 80 1.02 0.05 1.07145 0.02786 0.010789

5, 10, 20, 40 2.22 1.21 1.07804 0.003375 0.004199

6, 9, 12, 15 2.49 1.52 1.0376 0.05920 0.044639

Density at x = 0.25 True errors less than GCI in four cases Uncertainty estimates larger than GCI in

four cases Estimate lower than error in one case

36

Position of Lagrangian tracer x0 = 0.60 True errors less than for GCI, except one

case Uncertainty usually less than GCI Two uncertainty estimates are lower than

the true errorExact value:x= 0.732558

velocitydensity

Sod results using RMR

Sequence(NX)

Convergence Rate

Rate Uncertainty



True Error

20, 40, 80, 160 1.62 ±0.61 0.732821 ±0.0001170 -0.0002626

15, 30, 60, 120 1.67 ±0.93 0.732906 ±0.0005951 -0.0003476

25, 30, 35, 40 1.75 ±0.74 0.733104 ±0.040798 -0.0005456

80, 90, 100, 110 1.71 ±0.65 0.732853 ±0.00017428 -0.0002946

80, 160, 320, 640 1.09 ±0.57 0.732574 ±0.00006559 -0.0000156

85, 170, 340, 680 1.82 ±0.55 0.732595 ±0.0001470 -0.0000366

37

Density at x = 0.45 True errors less than GCI in half the

cases Uncertainties less than GCI in half the

cases Uncertainty bounds error


Sod results using RMR

Sequence(NX)

Convergence Rate

Rate Uncertainty



True Error

20, 40, 80, 160 1.19 ±2.24 0.49161 ±0.00668543 0.0026658

15, 30, 60, 120 1.56 ±1.32 0.487928 ±0.00943468 0.0063478

25, 30, 35, 40 1.59 ±0.98 0.455009 ±0.486196 0.0392668

80, 90, 100, 110 1.67 ±0.71 0.499523 ±0.0571008 -0.00524719

80, 160, 320, 640 1.26 ±1.80 0.493892 ±0.0012934 0.0003838

85, 170, 340, 680 1.72 ±0.64 0.496217 ±0.0029657 -0.00194118

38

RMR models for Sod density, x=0.45

Points are clustered Fits are ill-conditioned Large spread as h->0

Sequence:u80=0.4894 u90=0.499668u100=0.508325u110=0.498542

RMR estimate:u=0.499523

Exact value:u=0.49427581

Info gap analysis is a non-probabilistic approach to UQ that offers a method ‐for supporting model-based decisions under severe uncertainty. The emphasis is on decision support, not on “get the/an answer.” An info-gap model provides a weaker characterization of uncertainty

than probability-based models (e.g., there’s no PDF).

Info-Gap Approach toSolution Verification

For an info-gap analysis, three elements are required:

② Performance requirement: this is the error estimate for the system, with a mesh-resolved solution estimate based on the computed results.

① A model of the system: in this case, this is the standard error ansatz.

③ An info-gap model of uncertainty: a family of nested sets involving the uncertain parameters, which determines a non-probabilistic quantification of the uncertainty in the model.

An “info gap” is a disparity between: (1) what ‐ is known and (2) what needs to be known in order to make a decision of interest.

More info: http://info-gap.com

Credit: Kamm, Witkowski, Rider, Trucano, Ben-Haim, SAND2012-7798C

2nd orderu(2) = 0.1348 ± E

Info-Gap Solution VerificationAnalysis of a “toy” problem The resulting uncertainty-vs.-e surface is the final outcome of this analysis.

Solution Error Example: the ODE du/dt = –u(t), u(0) = 1, and two integration schemes, 1st order and 2nd order, to evaluate u(t=2) = 0.1353.

With three values, the standard approaches are:

u(1) = 0.1216 ± 0.0178 u(2) = 0.1358 ± 0.0016

u(1) = 0.1216 ± 0.0337u(2) = 0.1358 ± 0.0017

Grid Convergence Index

Xing & Stern

Info-gap gives quantitative behavior of uncertainty:

1st orderu(1) = 0.1370 ± E

e = “Horizon of Uncertainty”


Info-Gap Solution VerificationAnalysis of a “real” problem Example: Plate penetration problem run with SIERRA SM.

We analyze the final velocity of the penetrator at t = 8 ms.

Obviously, there is no exact solution for this problem.

Coarse Mesh

Al plate

Vz = -6.1 m s-1

Fine Mesh12.7 mm

12.7 mmt = 8 ms

Calculations performed by Nicole Breivik (Sandia)

Grid Convergence Index

Correction Factor or Xing & Stern

Standard approaches using two computed results:

Requires ≥ 3 results

There is very little dependence on properties of the estimated mesh-resolved solution,

Info-gap results:1st order

through a free parameter p* .


45

A brief history of solution verification Analysis of mesh-based methods established

the power-law form of the truncation error. This is the theoretical foundation upon which numerical methods are based

Roache inverted this relation to estimate the numerical error, given solutions at different resolutions. Since then, others have contributed using the same basic approach

This approach is being reexamined at a deeper level and alternatives are being proposed

e

RMR

ec

GCI

efec

Info-Gap

ec

Comparison of Solution Verification Techniques

Documents

Transcript of Comparison of Solution Verification Techniques