04/19/23 1

Summary of Experimental Uncertainty Assessment

Methodology

F. Stern, M. Muste, M-L. Beninati, W.E. EichingerF. Stern, M. Muste, M-L. Beninati, W.E. Eichinger

Table of Contents IntroductionIntroduction

Test Design PhilosophyTest Design Philosophy

DefinitionsDefinitions

Measurement Systems, Data-Reduction Equations, Measurement Systems, Data-Reduction Equations,

and Error Sourcesand Error Sources

Uncertainty Propagation EquationUncertainty Propagation Equation

Uncertainty Equations for Single and Multiple Uncertainty Equations for Single and Multiple

TestsTests

Implementation & RecommendationsImplementation & Recommendations

Introduction

Experiments are an essential and integral tool for Experiments are an essential and integral tool for engineering and scienceengineering and science

Experimentation:Experimentation: procedure for testing or determination of a procedure for testing or determination of a truth, principle, or effecttruth, principle, or effect

True values are seldom known and experiments have errors True values are seldom known and experiments have errors due to instruments, data acquisition, data reduction, and due to instruments, data acquisition, data reduction, and environmental effectsenvironmental effects

Therefore, determination of truth requires estimates for Therefore, determination of truth requires estimates for experimental errors, i.e., uncertaintiesexperimental errors, i.e., uncertainties

Uncertainty estimates are imperative for risk assessments in Uncertainty estimates are imperative for risk assessments in design both when using data directly or in calibrating and/or design both when using data directly or in calibrating and/or validating simulation methodsvalidating simulation methods

Introduction

Uncertainty analysis (UA): rigorous methodology for Uncertainty analysis (UA): rigorous methodology for uncertainty assessment using statistical and engineering uncertainty assessment using statistical and engineering conceptsconcepts

ASME (1998) and AIAA (1999) standards are the most ASME (1998) and AIAA (1999) standards are the most recent updates of UA methodologies, which are recent updates of UA methodologies, which are internationally recognized internationally recognized

Presentation purpose: to provide summary of EFD UA Presentation purpose: to provide summary of EFD UA methodology accessible and suitable for student and methodology accessible and suitable for student and faculty use both in classroom and research laboratories faculty use both in classroom and research laboratories

Test design philosophy Purposes for experiments: Purposes for experiments:

Science & technologyScience & technology Research & developmentResearch & development Design, test, and product liability and acceptanceDesign, test, and product liability and acceptance InstructionInstruction

Type of tests:Type of tests: Small- scale laboratorySmall- scale laboratory Large-scale TT, WTLarge-scale TT, WT In-situ experimentsIn-situ experiments

Examples of fluids engineering tests:Examples of fluids engineering tests: Theoretical model formulationTheoretical model formulation Benchmark data for standardized testing and evaluation of facility biasesBenchmark data for standardized testing and evaluation of facility biases Simulation validationSimulation validation Instrumentation calibrationInstrumentation calibration Design optimization and analysisDesign optimization and analysis Product liability and acceptanceProduct liability and acceptance

Test design philosophy

Decisions on conducting experiments: governed Decisions on conducting experiments: governed by the ability of the expected test outcome to by the ability of the expected test outcome to achieve the test objectives within allowable achieve the test objectives within allowable uncertainties uncertainties

Integration of UA into all test phases should be a Integration of UA into all test phases should be a key part of entire experimental programkey part of entire experimental program Test descriptionTest description Determination of error sourcesDetermination of error sources Estimation of uncertaintyEstimation of uncertainty Documentation of the resultsDocumentation of the results

Test design philosophy D E F IN E P U R P O S E O F T E S T A N D

R E S U LT S U N C E R TA IN T Y R E Q U IR E M E N T S

U N C E R TA IN T YA C C E P TA B L E ?

IM P R O V E M E N TP O S S IB L E ?

D E T E R M IN E E R R O R S O U R C E SA F F E C T IN G R E S U LT S

Y E SN O

N O

Y E S Y E S

Y E S

N O

S E LE C T U N C E R TA IN T Y M E T H O D

E S T IM AT E E F F E C T O FT H E E R R O R S O N R E S U LT S

- M O D E L C O N F IG U R AT IO N S (S )- T E S T T E C H N IQ U E (S )- M E A S U R E M E N T S R E Q U IR E D- S P E C IF IC IN S T R U M E N TAT IO N- C O R R E C T IO N S T O B E A P P L IE D

- D E S IR E D PA R A M E T E R S (C , C , ... .)D R

D E S IG N T H E T E S T

- R E F E R E N C E C O N D IT IO N- P R E C IS IO N L IM IT- B IA S L IM IT- T O TA L U N C E R TA IN T Y

D O C U M E N T R E S U LT S

N O T E S T

C O N T IN U E T E S T

IM P L E M E N T T E S T

S O LV E P R O B LE M

R E S U LT SA C C E P TA B L E ?

M E A SU R E-M E N T

S YS TE MP RO BLE M ?

N O

P U R P O S EA C H IE V E D ?

Y E S

N O

S TA R T T E S T

E S T IM AT EA C T U A L D ATAU N C E R TA IN T Y

Definitions

Accuracy:Accuracy: closeness of agreement between closeness of agreement between measured and true valuemeasured and true value

Error:Error: difference between measured and true valuedifference between measured and true value

Uncertainties (Uncertainties (UU):): estimate of errors in estimate of errors in measurements of individual variables measurements of individual variables XXii ( (UUxxii) or ) or

results (results (UUrr) obtained by combining ) obtained by combining UUxxii

Estimates of Estimates of UU made at made at 95% confidence level95% confidence level

Definitions

= to tal e rro r = b ias e rro r = p recis ion error

M A G N ITU D E O F XF

RE

QU

EN

CY

OF

OC

CU

RR

EN

CE

X

X

X

true

X true

k

k+ 1 k

k+ 1

k+ 1XX k

(a ) tw o re a d in g s

(b ) in f in ite n u m b e r o f re a d in g s

Bias errorBias error : fixed, : fixed, systematicsystematic

Bias limitBias limit B: estimate B: estimate of of

Precision errorPrecision error : : randomrandom

Precision limit Precision limit PP: : estimate of estimate of

Total error:Total error:

Measurement systems, data reduction equations, & error sources Measurement systems for individual variables Measurement systems for individual variables XXii: :

instrumentation, data acquisition and reduction procedures, instrumentation, data acquisition and reduction procedures, and operational environment (laboratory, large-scale and operational environment (laboratory, large-scale facility, in situ) often including scale modelsfacility, in situ) often including scale models

Results expressed through data-reduction equationsResults expressed through data-reduction equations

rr = = rr((XX11, , XX22, , XX33,…, ,…, XXjj))

Estimates of errors are meaningful only when considered Estimates of errors are meaningful only when considered in the context of the process leading to the value of the in the context of the process leading to the value of the quantity under considerationquantity under consideration

Identification and quantification of error sources require Identification and quantification of error sources require considerations of: considerations of:

Steps used in the process to obtain the measurement of the quantitySteps used in the process to obtain the measurement of the quantity The environment in which the steps were accomplishedThe environment in which the steps were accomplished

Measurement systems and data reduction equations Block diagram showing elemental error sources, individual Block diagram showing elemental error sources, individual

measurement systems, measurement of individual variables, measurement systems, measurement of individual variables, data reduction equations, and experimental resultsdata reduction equations, and experimental results

r = r (X , X ,......, X ) 1 2 J

1 2 J

M EASUREM ENTOF INDIVIDUALVARIABLES

INDIVIDUALM EASUREM ENTSYSTEMS

ELEM ENTALERROR SOUR CES

DATA REDUCTIONEQUATION

EXPERIM EN TALRESULT

XB , P

1

1 1

XB , P

2

2 2

XB , P

J

J J

rB , P

r r

Error sources Estimation assumptions: 95% confidence level, Estimation assumptions: 95% confidence level,

large-sample, statistical parent distributionlarge-sample, statistical parent distribution

MODEL FIDELITY ANDTEST SETUP:- As built geometry- Hydrodynamic deformation- Surface fin ish- Model positioning

TEST ENVIRONMENT:- Calibration versus test- Spatial/temporal variations o f the flow- Sensor insta llation/location- Wall inte rfe rence- F luid and facility conditions

DATA ACQUISITION ANDREDUC TION:- Sampling, filtering, and statistics- Curve fits- Calibrations

SIMULATION TECHNIQUES:- Instrumentation interference- Scale effects

CONTR IBUTIONSTO ESTIMATEDUNCERTAINITY

Uncertainty propagation equation Bias and precision errors in the measurement of Bias and precision errors in the measurement of XXii

propagate through the data reduction equation propagate through the data reduction equation rr = = rr((XX11, , XX22, ,

XX33,…, ,…, XXjj) resulting in bias and precision errors in the ) resulting in bias and precision errors in the

experimental result experimental result rr

A small error (A small error (XiXi) in the measured variable leads to a ) in the measured variable leads to a

small error in the result (small error in the result (rr) that can be approximated using ) that can be approximated using

Taylor series expansion of Taylor series expansion of rr((XXii) about ) about rrtruetrue((XXii) as) as

The derivative is referred to as sensitivity coefficient. The The derivative is referred to as sensitivity coefficient. The larger the derivative/slope, the more sensitive the value of larger the derivative/slope, the more sensitive the value of the result is to a small error in a measured variablethe result is to a small error in a measured variable

iXitrueir dX

drXrXr

i )()(

Uncertainty propagation equation Overview given for derivation of equation describing the error Overview given for derivation of equation describing the error

propagation with attention to assumptions and approximations used to propagation with attention to assumptions and approximations used to obtain final uncertainty equation applicable for single and multiple obtain final uncertainty equation applicable for single and multiple tests tests

Two variables, Two variables, kkth set of measurements (th set of measurements (xxkk, , yykk))

x k

k

k k k

ktrue true

true

k

k

k

k

k

k

r r x , y )= (

x

r

x

r

r

x

x

r

yy

y

y

y

),( yxrr

kk xxtruek xx

kk yytruek yy

2Ryyy

rxx

x

rrr truektruektruek

kkkkk yyyxxxtruekr rr

The total error in the The total error in the kkth determination of th determination of rr

yrxr yx ;sensitivity coefficientssensitivity coefficients

Uncertainty propagation equation We would like to know the distribution of We would like to know the distribution of r r (called the parent (called the parent

distribution) for a large number of determinations of the result distribution) for a large number of determinations of the result rr

A measure of the parent distribution is its variance defined as A measure of the parent distribution is its variance defined as

N

kr

N kr N 1

22 1lim

yxxxyxyxr yxyxyxyx 22 222222222

Substituting (1) into (2), taking the limit for N approaching infinity, Substituting (1) into (2), taking the limit for N approaching infinity,

using definitions of variances similar to equation (2) for using definitions of variances similar to equation (2) for ’s and ’s and ’s’s and and their correlation, and assuming no correlated bias/precision errorstheir correlation, and assuming no correlated bias/precision errors

’’s in equation (3) are not known; estimates for them must be made s in equation (3) are not known; estimates for them must be made

Uncertainty propagation equation Defining

estimate for

estimates for the variances and covariances (correlated bias errors) of

the bias error distributions

estimates for the variances and covariances ( correlated precision

errors) of the precision error distributions

xyyxyyxxxyyxyyxxc SSSbbbu 22 222222222

cr KuU

equation (3) can be written asequation (3) can be written as

To obtain uncertainty To obtain uncertainty UUrr at a specified confidence level ( at a specified confidence level (CC%), a %), a

coverage factor (coverage factor (KK) must be used for ) must be used for uucc::

For normal distribution, For normal distribution, K K is theis the t t value from the Studentvalue from the Student t t distribution.distribution.

For For N N 10, 10, t t = 2 for 95% confidence level = 2 for 95% confidence level

2cu 2

r222 ,, xyyx bbb

xyywx SSS ,, 2

Valid for any type of error Valid for any type of error distributiondistribution

Uncertainty propagation equation Generalization for Generalization for JJ variables in a result variables in a result rr = = rr((XX11, , XX22, , XX33,…, ,…, XXjj))

J

i

J

i

J

ikikkiii

J

i

J

i

J

ikikkiiir PP BB = U

1

1

1 1

22

1

1

1 1

222 22

ii X

r

AUDCAU

DC DD ,,,

21 2

222

222

22

2

222

1

22

1

222

AAD

UUDD

DDD

J

iii

J

iiiC

PBA

CPB

U

CPB

CPB

D

C

PBUD

sensitivity coefficientssensitivity coefficients

Example:Example:

Uncertainty equations for single and multiple tests

Measurements can be made in several ways:Measurements can be made in several ways:

Single test (for complex or expensive experiments): one set Single test (for complex or expensive experiments): one set of measurements (of measurements (XX11, , XX22, …, , …, XXjj) for ) for rr

According to the present methodology, a test is considered a single test if the entire According to the present methodology, a test is considered a single test if the entire test is performed only once, even if the measurements of one or more variables are test is performed only once, even if the measurements of one or more variables are made from many samples (e.g., LDV velocity measurements)made from many samples (e.g., LDV velocity measurements)

Multiple tests (ideal situations): many sets of Multiple tests (ideal situations): many sets of measurements (measurements (XX11, , XX22, …, , …, XXjj) for ) for rr at a fixed test condition at a fixed test condition

with the same measurement systemwith the same measurement system

Uncertainty equations for single and multiple tests

The total uncertainty of the result The total uncertainty of the result

P + B = U 2r

2rr

2

BBr r : same estimation procedure for single and : same estimation procedure for single and

multiple testsmultiple tests

PPr r : determined differently for single and multiple : determined differently for single and multiple

teststests

Uncertainty equations for single and multiple tests: bias limits

BBrr : :

J

i

J

i

J

ikikkiiir BBB

1

1

1 1

222 2

ii X

r

BBii: estimate of calibration, data acquisition, data reduction, conceptual : estimate of calibration, data acquisition, data reduction, conceptual

bias errors for bias errors for XXi.i.. Within each category, there may be several elemental . Within each category, there may be several elemental

sources of bias. If for variable sources of bias. If for variable XXii there are there are JJ significant elemental bias significant elemental bias

errors [estimated as errors [estimated as (B(Bii))11, , (B(Bii))2,2, … … (B(Bii))JJ], the bias limit for ], the bias limit for XXii is calculated as is calculated as

BBikik: estimate of correlated bias limits for : estimate of correlated bias limits for XXi i and and XXkk

L

kiik BBB1

Sensitivity coefficientsSensitivity coefficients

2

1

2

k

J

kii BB

Uncertainty equations for single test: precision limits

Precision limit of the result (end to end):Precision limit of the result (end to end):rr tSP

tt: coverage factor (: coverage factor (t t = 2 for = 2 for NN > 10) > 10)

SSrr: the standard deviation for the : the standard deviation for the NN readings of the result. readings of the result. SSrr must be must be

determined from determined from NN readings over an appropriate/sufficient time interval readings over an appropriate/sufficient time interval

Precision limit of the result (individual variables):Precision limit of the result (individual variables):

)P ( P 2ii

J

=1ir

iii StP the precision limits for the precision limits for XXii

Often is the case that the time interval is inappropriate/insufficient and Often is the case that the time interval is inappropriate/insufficient and PPii’s ’s

or or PPrr’s must be estimated based on previous readings or best available ’s must be estimated based on previous readings or best available

informationinformation

Uncertainty equations for multiple tests: precision limits

The average result:The average result:

tt: coverage factor (: coverage factor (t t = 2 for = 2 for NN > 10) > 10)

: standard deviation for : standard deviation for MM readings of the result readings of the result

Precision limit of the result (end to end):Precision limit of the result (end to end):

M

kkrM

r1

1

M

tSP r

r 2/1

1

2

1

M

k

kr M

rrS

The total uncertainty for the average result:The total uncertainty for the average result:

Alternatively can be determined by RSS of the Alternatively can be determined by RSS of the precision limits of the individual variablesprecision limits of the individual variables

rS

rP

22222 2 MSBPBU rrrrr

Implementation Define purpose of the testDefine purpose of the test

Determine data reduction equation: Determine data reduction equation: rr = = rr((XX11, , XX22, …, , …, XXjj) )

Construct the block diagramConstruct the block diagram

Construct data-stream diagrams from sensor to resultConstruct data-stream diagrams from sensor to result

Identify, prioritize, and estimate bias limits at individual variable levelIdentify, prioritize, and estimate bias limits at individual variable level Uncertainty sources smaller than 1/4 or 1/5 of the largest sources are neglectedUncertainty sources smaller than 1/4 or 1/5 of the largest sources are neglected

Estimate precision limits (end-to-end procedure recommended)Estimate precision limits (end-to-end procedure recommended) Computed precision limits are only applicable for the random error sources that were Computed precision limits are only applicable for the random error sources that were

“active” during the repeated measurements“active” during the repeated measurements Ideally Ideally MM 10, however, often this is no the case and for 10, however, often this is no the case and for MM < 10, a coverage factor < 10, a coverage factor tt

= 2 is still permissible if the bias and precision limits have similar magnitude.= 2 is still permissible if the bias and precision limits have similar magnitude. If unacceptably large If unacceptably large PP’s are involved, the elemental error sources contributions must ’s are involved, the elemental error sources contributions must

be examined to see which need to be (or can be) improvedbe examined to see which need to be (or can be) improved

Calculate total uncertainty using equation (4)Calculate total uncertainty using equation (4)

For each For each r,r, report total uncertainty and bias and precision limits report total uncertainty and bias and precision limits

Recommendations

Recognize that uncertainty depends on entire testing process and that Recognize that uncertainty depends on entire testing process and that any changes in the process can significantly affect the uncertainty of any changes in the process can significantly affect the uncertainty of the test resultsthe test results

Integrate uncertainty assessment methodology into all phases of the Integrate uncertainty assessment methodology into all phases of the testing process (design, planning, calibration, execution and post-test testing process (design, planning, calibration, execution and post-test analyses)analyses)

Simplify analyses by using prior knowledge (e.g., data base), Simplify analyses by using prior knowledge (e.g., data base), concentrate on dominant error sources and use end-to-end calibrations concentrate on dominant error sources and use end-to-end calibrations and/or bias and precision limit estimationand/or bias and precision limit estimation

Document: Document: test design, measurement systems, and data streams in block diagramstest design, measurement systems, and data streams in block diagrams equipment and procedures usedequipment and procedures used error sources considerederror sources considered all estimates for bias and precision limits and the methods used in their estimation all estimates for bias and precision limits and the methods used in their estimation

(e.g., manufacturers specifications, comparisons against standards, experience, etc.)(e.g., manufacturers specifications, comparisons against standards, experience, etc.) detailed uncertainty assessment methodology and actual data uncertainty estimatesdetailed uncertainty assessment methodology and actual data uncertainty estimates

References AIAA, 1999, “Assessment of Wind Tunnel Data Uncertainty,” AIAA AIAA, 1999, “Assessment of Wind Tunnel Data Uncertainty,” AIAA

S-071A-1999.S-071A-1999. ASME, 1998, “Test Uncertainty,” ASME PTC 19.1-1998.ASME, 1998, “Test Uncertainty,” ASME PTC 19.1-1998. ANSI/ASME, 1985, “Measurement Uncertainty: Part 1, Instrument ANSI/ASME, 1985, “Measurement Uncertainty: Part 1, Instrument

and Apparatus,” ANSI/ASME PTC 19.I-1985.and Apparatus,” ANSI/ASME PTC 19.I-1985. Coleman, H.W. and Steele, W.G., 1999, Coleman, H.W. and Steele, W.G., 1999, Experimentation and Experimentation and

Uncertainty Analysis for EngineersUncertainty Analysis for Engineers, 2, 2ndnd Edition, John Wiley & Sons, Edition, John Wiley & Sons, Inc., New York, NY.Inc., New York, NY.

Coleman, H.W. and Steele, W.G., 1995, “Engineering Application of Coleman, H.W. and Steele, W.G., 1995, “Engineering Application of Experimental Uncertainty Analysis,” AIAA Journal, Vol. 33, No.10, Experimental Uncertainty Analysis,” AIAA Journal, Vol. 33, No.10, pp. 1888 – 1896.pp. 1888 – 1896.

ISO, 1993, “ISO, 1993, “Guide to the Expression of Uncertainty in Guide to the Expression of Uncertainty in MeasurementMeasurement,", 1st edition, ISBN 92-67-10188-9.,", 1st edition, ISBN 92-67-10188-9.

ITTC, 1999, ITTC, 1999, Proceedings 22Proceedings 22ndnd International Towing Tank Conference International Towing Tank Conference, , “Resistance Committee Report,” Seoul Korea and Shanghai China.“Resistance Committee Report,” Seoul Korea and Shanghai China.