Uncertainty Basics

52
An Introduction to Expressing Uncertainty in Measurement PALCAN Team Leaders’ Seminar Ottawa, Nov. 2002 Mike Ouellette, CLAS Technical Advisor 2002-10-25 M. Ouellette 1 National Research Council Canada Conseil national de recherches Canada Institute for National Measurement Standards Institut des étalons nationaux de mesure

Transcript of Uncertainty Basics

Page 1: Uncertainty Basics

An Introduction to Expressing Uncertainty in Measurement

PALCAN Team Leaders’ SeminarOttawa, Nov. 2002

Mike Ouellette, CLAS Technical Advisor

2002-10-25 M. Ouellette 1

National ResearchCouncil Canada

Conseil nationalde recherches Canada

Institute for NationalMeasurement Standards

Institut des étalonsnationaux de mesure

Page 2: Uncertainty Basics

2002-10-25M. Ouellette 2

Overview

• Why the Big Deal about Uncertainty?

• Background Stats: The Basics -- Nothing Fancy- Minimal stats jargon- Minimal algebraic notations- No system modeling- No calculus- Promise!

• How to apply it (on the back of an envelope)

• A Worked Example

Page 3: Uncertainty Basics

2002-10-25M. Ouellette 3

Why the Big Deal?

• 17025

• VIM (re. Traceability)

No stated uncertainty = No Traceability !

Page 4: Uncertainty Basics

2002-10-25M. Ouellette 4

Don’t get Sucked in!

• ESTIMATE uncertainty only to the extent that you need, don’t make a science of it.

• Accept that you’ll never be certain about uncertainty.

• Focus mostly on the “biggies”; don’t sweat the small stuff (much).

Page 5: Uncertainty Basics

2002-10-25M. Ouellette 5

The Background Stats:The Standard Deviation

• Repeat measurements aren’t identical.

• Stdev tells us how widely they are dispersed about the mean.

• Approx. 2/3 (i.e., 68%) of all readings fall within 1 stdev of the mean.

-1 S

tdev

+1

Std

ev

-3 -2 -1 0 1 2 3

Differences from the mean

68.3%

Page 6: Uncertainty Basics

2002-10-25M. Ouellette 6

The Background Stats:The Standard Deviation

• Approx. 95.5% of all readings fall within 2 stdev of the mean.

+2S

tdev

-2S

tdev

-3 -2 -1 0 1 2 3

Differences from the mean

95.5%

Page 7: Uncertainty Basics

1. Get average Reading,X X - average (X-average)^2

2. Deviations from 1,000,000,043 -7 49average 1,000,000,055 5 25

1,000,000,055 5 253. Square those 1,000,000,051 1 1deviations 1,000,000,058 8 64

1,000,000,043 -7 494. Sum the squares 1,000,000,045 -5 255. Divide by (n-1) 1,000,000,045 -5 256. Take sqr root 1,000,000,057 7 49

1,000,000,048 -2 4=average() 1,000,000,050 Sum()= 316

=Count(), n 10Excel Stdev Fnc:15.08 n - 1 9Nominal: Sum / (n-1) 35.111.00E+09 Stdev= sqrt[Sum/(n-1)] 5.93

To Calculate Standard Deviation

Step 1

Step 2 Step 3

Step 4

Step 5

Step 6

Page 8: Uncertainty Basics

1. Get average Reading,X X - average (X-average)^2

2. Deviations from 1,000,000,043 -7 49average 1,000,000,055 5 25

1,000,000,055 5 253. Square those 1,000,000,051 1 1deviations 1,000,000,058 8 64

1,000,000,043 -7 494. Sum the squares 1,000,000,045 -5 255. Divide by (n-1) 1,000,000,045 -5 256. Take sqr root 1,000,000,057 7 49

1,000,000,048 -2 4=average() 1,000,000,050 Sum()= 316

=Count(), n 10Excel Stdev Fnc:15.08 n - 1 9Error in Stdev is: 155% Sum / (n-1) 35.11

Nominal = 1.00E+09 Stdev= sqrt[Sum/(n-1)] 5.93try 4.0e8, 4.2e9, 10e12, 11e12

Step 1

Step 2 Step 3

Step 4

Step 5

Step 6

Page 9: Uncertainty Basics

2002-10-25M. Ouellette 9

Experimental Std Deviation of the Mean

• The stdev describes the spread of values in an individual set of measurements. What if we took several different sets of measurements?

• The mean of each set of measurements would vary.

• The spread of the means is given by the experimental std deviation of the mean (stdm).

• To predict stdm from a set of n replicate measurements, divide the stdev by (n).

Stdm = stdev / (n)

Page 10: Uncertainty Basics

2002-10-25M. Ouellette 10

Experimental Std Deviation of the Mean: An Example

• If standard deviation, Stdev, is 5.93, and

• if number of replicate measurements, n, is 10, then

• experimental standard deviation of the mean, stdm, is

Stdm = Stdev / sqrt (n)

= 5.93 / sqrt (10)

= 1.88

Page 11: Uncertainty Basics

2002-10-25M. Ouellette 11

Experimental Std Deviation of the Mean: So what?

• The stdm feeds directly into our uncertainty budget whenever we estimate uncertainty in the mean of a set of random repeated measurements.

Very Useful!More on that later

Page 12: Uncertainty Basics

2002-10-25M. Ouellette 12

Distributions of Measurements

• The spread of a set of measurements can take on different frequency distributions. Examples:

Page 13: Uncertainty Basics

2002-10-25M. Ouellette 13

Distributions of Measurements: Normal Distribution (Gaussian)

• Most values fall near the mean.

• Progressively fewer values falling further from the mean.

• Very common distribution in nature.

Page 14: Uncertainty Basics

2002-10-25M. Ouellette 14

Distributions of Measurements: Uniform Distribution (Rectangular)

• Measurements distributed equally across the interval.

• Reasonably good choice when you don’t have a clue!

Page 15: Uncertainty Basics

2002-10-25M. Ouellette 15

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Uncertainty in their calibration Sensitivity to transportation & handling

Long term driftDesign issues (such as unequal lengths of equal arm balances)

Stability during measurement ParallaxesResolution & quality of their scales Interpolation between calibration pointsLinearity Reading systemDigitization Transporation

Reference Standards & Measurement Equipment; e.g…..

Not a complete list. Not all necessarily apply. Ditto for subsequent slides.

Make a List of Contributors. Consider ...

Page 16: Uncertainty Basics

2002-10-25M. Ouellette 16

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Absolute temperature ContaminationTime variance in temperature IlluminationSpatial gradiant in temperature Air pressure, composition, flowHeat radiation from operator, etc. GravityHumidity Electromagnetic interferenceNoise, Vibration Transients in power supply

Environmental Conditions; e.g…..

Page 17: Uncertainty Basics

2002-10-25M. Ouellette 17

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Warming up Loading effectsCosine errors Cabling, shielding, filteringOptical aperture Stiffness/rigidity of mechanical systemsParasitic voltages Properties of measurement probesCurrent leakage

Measurement Setup; e.g…..

Page 18: Uncertainty Basics

2002-10-25M. Ouellette 18

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Sensitivity to stresses of measurement CleanlinessSurface roughness Internal strengthConductivity Distortion during measurement

Weight, size, shapeSelf heating (e.g., current measurement)

Magnetism No. of terminalsStability Orientation

Measurement Object; e.g…..

Page 19: Uncertainty Basics

2002-10-25M. Ouellette 19

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Repeatability, conditioning Choice of reference & apparatusNo. & order of measurements Clamping, fixturing, probingDuration of measurements Drift checkChoice of principle of measurements Reversal measurementsMagnetism Multiple redundancy checksAlignment Strategy

Measurement Process; e.g…..

Page 20: Uncertainty Basics

2002-10-25M. Ouellette 20

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Rounding FilteringAlgorithms InterpolationNo. of signif. digits in calculat'n ExtrapolationSampling Outlier handling

Software & Calculations; e.g…..

Page 21: Uncertainty Basics

2002-10-25M. Ouellette 21

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

E.g., definition of "Diameter" for dimensional measurements of objects that are not perfectly round.

Definition of the Measurement Characteristic; e.g…..

D= 0.5 (Max + Min) ?

D= Mean of many measurements?

D= Other? (e.g., Bullets --> Max diameter?)

Page 22: Uncertainty Basics

2002-10-25M. Ouellette 22

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Uncertainty in knowledge of the physical values used

Various properties of the working, measuring instrument

Temperature coefficient Various properties of ambient airPower coefficient Local force of gravity

Physical Constants and Conversion Factors; e.g…..

Page 23: Uncertainty Basics

2002-10-25M. Ouellette 23

Estimating Uncertainty Step 1:Identify the Contributors to Uncertainty

Differences in judgements of different operators. Metrologist Effects; e.g…..

Note: Shouldn’t need to include uncertainty due to “Operator Error.” This is an aberration that should be corrected before measurement.

Has System Stabilised?

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 5 10 15 20

Time of Measurement, Minutes

Me

as

ure

d V

alu

e

How would you integrate shoulder peaks, below?

Page 24: Uncertainty Basics

2002-10-25M. Ouellette 24

Estimating Uncertainty Step 1:Summary of Some Contributors to Uncertainty

• Reference Standards & Measurement Equipment

• Environmental Conditions

• Measurement Setup

• Measurement Object

• Measurement Process

• Software & Calculations

• Definition of the Measurement Characteristic

• Physical Constant & Conversion Factors

• Metrologist Effects

Page 25: Uncertainty Basics

2002-10-25M. Ouellette 25

Step 2: Decide on the Uncertainty Units

Units may be:

• ppm or % of result

• units of measurement

• other

Generally doesn’t matter what you pick; just be consistent: Apples + Apples = Apples.

Everything gets boiled down to a standard deviation of the mean or equivalent (called standard uncertainty).

Page 26: Uncertainty Basics

2002-10-25M. Ouellette 26

Step 3: Estimate the Magnitude of the Uncertainty Contributors 1) Type A, Normal

Type A contributors to uncertainty are those that you have statistical data for. Use this data if you have it.

E.g., For Standard Uncertainty in the mean of repeated measurements (preferably 10 or more) use stdm, the experimental standard deviation of the mean.

Page 27: Uncertainty Basics

2002-10-25M. Ouellette 27

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B

Type B contributors to uncertainty are those that you have no statistical data for; e.g.,

• Manufacturer’s specification

• Professional judgement

• Uncertainty in cal certificate for your reference standard

GUM: There’s “no substitute for critical thinking, intellectual honesty, and professional skill.”

Page 28: Uncertainty Basics

2002-10-25M. Ouellette 28

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B

Gather information from wherever we can.

Boil down the uncertainty info to a standard uncertainty.

But info doesn’t always come at level of confidence of 68.3%. It usually comes at a higher level of confidence. It’s called an Expanded Uncertainty when coming from a normal distribution.

Page 29: Uncertainty Basics

2002-10-25M. Ouellette 29

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Normal

For a normal distribution, no problem!

Divisor Level ofConfidence

0.676 50%

1 68.27%

1.645 90%

1.960 95%

2 95.45%

2.576 99%

3 99.73%

Simply divide the Expanded Uncertainty by a Divisor, depending upon the level of confidence at which the Expanded Uncertainty is given.

Page 30: Uncertainty Basics

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Normal

E.g., Cal certificate for our mass standard states:

“The uncertainty in the reported mass is 26 mg at a level of confidence of 95% assuming a normal distribution.”

The standard uncertainty is….

26 mg 1.960 = 13 mg

Divisor Level ofConfidence

1.960 95%

2 95.45%

2.576 99%

3 99.73%

Page 31: Uncertainty Basics

2002-10-25M. Ouellette 31

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Normal

E.g., Cal certificate for our resistor states:

“The expanded uncertainty in the reported resistance is 30 with a coverage factor, k, of 3.”

The standard uncertainty is….

30 3 = 10 Here, the “coverage factor” is another word for “divisor”.

Page 32: Uncertainty Basics

2002-10-25M. Ouellette 32

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Uniform

But not all distributions are normal.

Equal probability that true value is anywhere between interval a- to a+ .

Rectangular (I.e., Uniform) distributions also very common.

Page 33: Uncertainty Basics

2002-10-25M. Ouellette 33

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Uniform

For Rectangular (I.e., Uniform) distributions , the Standard Uncertainty, u, is the area shaded in grey.

Standard uncertainty, u, is

the complete interval, a, divided by 3 or ~1.73.

Page 34: Uncertainty Basics

2002-10-25M. Ouellette 34

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Uniform

E.g., Gauge readability (resolution).

If smallest subdivision is 5 measurement units, andif we can “eyeball” the gauge to within 1/2 a subdivision, then the gauge’s true value can be anywhere in the interval a = ± 5 / 2 = ± 2.5 with equal probability.

Standard Uncertainty, u, = ± 2.5 / 3 = ± 1.4 meas. unitsor, if you prefer , u = ± 5 / 12 = ± 1.4 meas. units.

0 105

Page 35: Uncertainty Basics

2002-10-25M. Ouellette 35

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Uniform

E.g., Ambient temperature

Approx. equal probability that temperature is anywhere between control limits a+ = 24°C and a- = 22°C.

Standard Uncertainty, u, = ± 1°C / 3 = ± 0.58°C

Control Limits = 23°C ± 1°C

a-

a+

TIME

TEM

PERA

TURE

Page 36: Uncertainty Basics

2002-10-25M. Ouellette 36

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Uniform

u = ± 0.58°C x (0.5 ppm per °C ) = 0.29 ppm of reading.

But remember that we need to express all uncertainties in the same units!

Control Limits = 23°C ± 1°C

a-

a+

TIME

TE

MP

ER

AT

UR

E

If we’re expressing uncertainty in ppm of reading of a resistor, and if temperature coefficient of resistor is 0.5 ppm/°C, then std uncertainty becomes:

u = ± 0.58°C

Page 37: Uncertainty Basics

2002-10-25M. Ouellette 37

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Unknown Distrib’n

Clueless about the distribution?

Standard Uncertainty, u, = ± (Max Interval width) / 3

Usually safe to treat it as a Rectangular

(Uniform) distribution

Page 38: Uncertainty Basics

2002-10-25M. Ouellette 38

Step 3: Estimate the Magnitude of the Uncertainty Contributors 2) Type B, Unknown Distrib’n

Standard Uncertainty, u, = ± 0.02 mm / 3 = ± 0.012 mm

E.g., An incomplete instrument specification:

“The anvils of the micrometer are certified to be parallel within ± 0.02 mm.”

• No coverage factor or level of confidence,

• No distribution --> is spec. based on a stdev?

Page 39: Uncertainty Basics

2002-10-25M. Ouellette 39

Step 4: Document, Document, Document!

Your uncertainty budget is a living document that is revised as your measurement process changes and as your understanding of it improves. Therefore...

Write down how you arrived at each of the uncertainty estimates that you have listed.For example of documenting, see last slide.

File this info with the uncertainty budget.

Page 40: Uncertainty Basics

2002-10-25M. Ouellette 40

Any questions so far?

?

Page 41: Uncertainty Basics

2002-10-25M. Ouellette 41

Recap on Steps 1 to 4

1. Identify the contributors to measurement uncertainty.

2. Decide on consistent uncertainty units.

3. Estimate magnitude of each uncertainty contributor, and express each as a standard uncertainty.

The hardest part is done!

4. Document the basis for your estimates.

Page 42: Uncertainty Basics

2002-10-25M. Ouellette 42

Step 5: Combine the standard uncertainties into one number

Use the RSS method:

• Take the square of each std uncertainty, u

• Add up the squares

• Take the square root of the sum

This gives combined standard uncertainty, uc, of your measurement, with ~68% confidence.

Page 43: Uncertainty Basics

2002-10-25M. Ouellette 43

Step 5: Combine the standard uncertainties into one number

E.g., std uncertainty, u, in ppm of reading:

1. Cal of ref std: 0.75 ppm

2. Long term drift of ref std: 1.16 ppm

3. Repeatability, n=5: 0.07 ppm

Combined std uncertainty, uc, =

sqrt ( 0.752 + 1.162 + 0.072 ) = 1.4 ppm

i.e., ~68% confident that true value is within± 1.4 ppm of the reported result.

Page 44: Uncertainty Basics

2002-10-25M. Ouellette 44

Step 6: Expand the combined the standard uncertaintyWhat if we want to report uncertainty with a different level of confidence rather than ~68% ?

No problem! Simply multiply combined std uncertainty, uc, by an appropriate coverage factor, k. The product is called Expanded Combined Uncertainty, Uc.

Coveragefactor, k

Level ofConfidence

0.676 50%

1 68.27%

1.645 90%

1.960 95%

2 95.45%

2.576 99%

3 99.73%

Page 45: Uncertainty Basics

Step 6: Expand the combined the standard uncertainty

Coveragefactor, k

Level ofConfidence

0.676 50%

1 68.27%

1.645 90%

1.960 95%

2 95.45%

2.576 99%

3 99.73%

Exercise: Combined std unc,uc, is 1.4 ppm of rdg;

We want to report uncertainty with a level of confidence of “approximately 95%.”

Expanded Combined Uncertainty,Uc, = 1.4 ppm x 2 = 2.8 ppm

Page 46: Uncertainty Basics

2002-10-25M. Ouellette 46

Step 7: Reality check

The preceding simplified approach is based upon several assumptions, including:

• Dominant contributor(s) to uncertainty are known with reasonable certainty;

- For Type A: Have enough data ( 10 points)- For Type B: No wild guessing

Page 47: Uncertainty Basics

2002-10-25M. Ouellette 47

Step 7: Reality check

So identify the dominant contributor(s).

1. Cal of ref std: 0.75 ppm

2. Long term drift of ref std: 1.16 ppm

3. Repeatability, n=5: 0.071 ppm

Dominant contributor is drift. If wild guess or only few data points, then get a more reliable (otherwise a more conservative) estimate if possible. Otherwise follow detailed steps in the GUM on degrees of freedom.

Page 48: Uncertainty Basics

2002-10-25M. Ouellette 48

Step 7: Reality check

1. Cal of ref std: 0.75 ppm

2. Long term drift of ref std: 1.12 ppm

3. Repeatability, n=5: 0.071 ppm

Small n (of 5) for repeatability is good enough because this contributor isn’t dominant.

Focus on the biggies: Long term drift & ref std cal.

Page 49: Uncertainty Basics

2002-10-25M. Ouellette 49

Step 8: Reporting the result

“The measured result is 10,000.051 ± 2.8 ppm for a level of confidence of approximately 95%, k=2.”

“The measured result is 10,000.051 ± 0.028 for a level of confidence of approximately 95%, k=2.”

Better...

Better yet...

“The measured result is 10,000.051 ± 0.028 . The reported uncertainty is expanded using a coverage factor k=2 for a level of confidence of approximately 95%, assuming a normal distribution.”

Not Recommended...

Page 50: Uncertainty Basics

2002-10-25M. Ouellette 50

Recap on all Steps

1. Identify the contributors to measurement uncertainty.

2. Decide on consistent uncertainty units.

3. Estimate magnitude of each uncertainty contributor, and express each as a standard uncertainty.

4. Document the basis for your estimates.

5. Combine the standard uncertainties.

6. Expand combined std uncertainty to represent desired confidence.

7. Reality check.

8. Report the result.

Page 51: Uncertainty Basics

Source of Uncertainty Value ppm

ProbabilityDistribution

DivisorStandard

Uncertainty ppm

Repeatability 0.071 Normal 1 0.071

Calibration uncertainty ofstandard resistor

1.5 Normal 2 0.75

Uncorrected drift of standardresistor since last calibration

2.0 Rectangular 3 1.155

Effect of temperature of oilbath

0.5 Rectangular 3 0.289

Effect of voltmeter onmeasurement of standard

resistor0.2 Rectangular 3 0.115

Effect of voltmeter onmeasurement of unknown

resistor0.2 Rectangular 3 0.115

Combined standarduncertainty, uc

Assumed normal(k=1)

1.418

Expanded combineduncertainty, Uc

Assumed normal(k=2)

2.9

A more Complete Example Uncertainty BudgetFor cal of a 10 k resistor by voltage intercomparison in oil

Page 52: Uncertainty Basics

A more Complete Example Uncertainty BudgetWhere the numbers came from

• Repeatability from 5 random repeat measurements in ppm from nominal:+10.4, +10.7, +10.6, +10.3, +10.5 --> stdm of 0.071 ppm.

• Cal of std resistor from cal certificate saying uncertainty in reported result is 1.5 ppm at a level of confidence of approximately 95% (k=2);1.5 ppm / 2 = 0.75 ppm.

• Drift of ref std is based upon several years of calibration history and comparisions with a check standard. A trend line was fitted to the historical data and was used to correct the ref std’s value at the date that it was used. The estimated uncertainty in the correction is 2 ppm based upon an analysis of the regression residuals.

• Temperature effects were estimated from the control limits of the stirred oil bath ( 0.2°C) and the manufacturer’s specs for the resistors’ temperature coefficient (worst case 2.5 ppm/°C). Bath homogeneity was measured and found to be negligible (< 0.05°C).

• Voltmeter effects are limited to linearity and resolution because it is used as a transfer device only. According to the mfr’s specs, the combined effects of linearity & resolution are 0.2 ppm in the range of measurement. The uncertainty is considered twice: once for each resistor. Go back to Step 4