Error analysis lecture 10
Transcript of Error analysis lecture 10
-
7/30/2019 Error analysis lecture 10
1/32
Physics 509 1
Physics 509: Error Propagation, and
the Meaning of Error Bars
Scott OserLecture #10
October 14, 2008
-
7/30/2019 Error analysis lecture 10
2/32
Physics 509 2
What is an error bar?Someone hands you a
plot like this. What dothe error bars indicate?
Answer: you can never
be sure, unless it'sspecified!
Most common: verticalerror bars indicate 1uncertainties.
Horizontal error bars canindicate uncertainty onX coordinate, or can
indicate binning.
Correlations unknown!
-
7/30/2019 Error analysis lecture 10
3/32
Physics 509 3
Relation of an error bar to PDF shape
The error bar on a plot ismost often meant torepresent the 1 uncertaintyon a data point. Bayesiansand frequentists will disagree
on what that means.
If data is distributed normallyaround true value, it's clear
what is intended:
exp[-(x-)2/22].
But for asymmetric
distributions, different thingsare sometimes meant ...
-
7/30/2019 Error analysis lecture 10
4/32
Physics 509 4
An error bar is a shorthand approximation to aPDF!
In an ideal Bayesian universe, error bars don't exist.Instead, everyone will use the full prior PDF and thedata to calculate the posterior PDF, and then report
the shape of that PDF (preferably as a graph ortable).
An error bar is really a shorthand way to parameterize
a PDF. Most often this means pretending the PDF isGaussian and reporting its mean and RMS.
Many sins with error bars come from assumingGaussian distributions when there aren't any.
-
7/30/2019 Error analysis lecture 10
5/32
Physics 509 5
An error bar as a confidence interval
Frequentist techniques don't directly answer the question of what theprobability is for a parameter to have a particular value. All you cancalculate is the probability of observing your data given a value of theparameter.The confidence interval construction is a dodge to get
around this. Starting point is thePDF for the estimator,for a fixed value of theparameter.
The estimator hasprobability 1 tofall in the white region.
-
7/30/2019 Error analysis lecture 10
6/32
Physics 509 6
Confidence interval construction
The confidence band isconstructed so that the averageprobability of
truelying in the
confidence interval is 1-.
This should not be interpretedto mean that there is, forexample, a 90% chance thatthe true value of is in theinterval (a,b) on any given trial.
Rather, it means that if you ranthe experiment 1000 times, andproduced 1000 confidenceintervals, then in 90% of these
hypothetical experiments (a,b)would contain the true value.
Obviously a very roundaboutway of speaking ...
-
7/30/2019 Error analysis lecture 10
7/32
Physics 509 7
The ln(L) rule
It is not trivial to construct properfrequentist confidence intervals.Most often an approximation isused: the confidence interval for asingle parameter is defined as therange in which ln(L
max)-ln(L)
-
7/30/2019 Error analysis lecture 10
8/32
Physics 509 8
Error-weighted averages
Suppose you have N independent measurements of a quantity.You average them. The proper error-weighted average is:
x = x i / i
2
1/i
2
Vx =1
1 / i2
If all of the uncertainties are equal, then this reduces to the simplearithmetic mean, with V() = V(x)/N.
-
7/30/2019 Error analysis lecture 10
9/32
Physics 509 9
Bayesian derivation of error-weightedaverages
Suppose you have N independent measurements of a quantity,distributed around the true value with Gaussian distributions.For flat prior on we get:
Vx =1
1 / i2
Pxi exp [12
x i i 2
]=exp
[12i
x i i 2
]It's easy to see that this has the form of a Gaussian. To find itspeak, set derivative with respect to equal to zero.
dP
d=exp [12i x i i
2
] [i x i i2 ]=0 =xi /i
2
1 / i2
Calculating the coefficient of2 in the exponent yields:
-
7/30/2019 Error analysis lecture 10
10/32
Physics 509 10
Averaging correlated measurements
We already saw how to average N independent measurement.What if there are correlations among measurements?
For the case of uncorrelated Gaussianly distributedmeasurements, finding the best fit value was equivalent tominimizing the chi-squared:
2=i x i i
2
In Bayesian language, this comes about because the PDF for isexp(-2/2). Because we know that this PDF must be Gaussian:
Pexp
[1
2
0
2
]then an easy way to find the 1 uncertainties on is to find the
values of for which 2 = 2min
+ 1.
-
7/30/2019 Error analysis lecture 10
11/32
Physics 509 11
Averaging correlated measurements II
The obvious generalization for correlated uncertainties is to formthe 2 including the covariance matrix:
2=
i
j
x ix jV1ij
We find the best value of by minimizing this 2 and can then findthe 1 uncertainties on by finding the values of for which2 = 2
min+ 1.
This is really parameter estimation with one variable.
The best-fit value is easy enough to find:
=
i , j
x j V1ij
i , j
V1ij
-
7/30/2019 Error analysis lecture 10
12/32
Physics 509 12
Averaging correlated measurements III
Recognizing that the 2 really just is the argument of anexponential defining a Gaussian PDF for ...
2=
i
j
x ix jV1ij
we can in fact read off the coefficient of2, which will be 1/V():
2=
1
i , j V1
ij
In general this can only be computed by inverting the matrix asfar as I know.
-
7/30/2019 Error analysis lecture 10
13/32
Physics 509 13
Averaging correlated measurements IVSuppose that we have N
correlated measurements. Eachhas some independent error=1 and a common error b thatraises or lowers them alltogether. (You would simulateby first picking a random valuefor b, then for eachmeasurement picking a newrandom value c with RMS and
writing out b+c.
Each curve shows how the erroron the average changes with N,for different values of b.
b=1b=0.5b=0.25b=0
-
7/30/2019 Error analysis lecture 10
14/32
Physics 509 14
Averaging correlated measurements: exampleConsider the following example, adapted from Cowan's book:
We measure an object's length with two rulers. Both are calibrated to beaccurate at T=T
0, but otherwise have a temperature dependency: true length y is
related to measured length by:
We assume that we know the ciand the uncertainties, which are Gaussian. We
measure L1, L
2, and T, and so calculate the object's true length y.
y i=Lici TT0
y i=Lici TT0
We wish to combine the measurements from the two rulers to get our best estimate of
the true length of the object.
-
7/30/2019 Error analysis lecture 10
15/32
Physics 509 15
Averaging correlated measurements: exampleWe start by forming the covariance matrix of the two measurements:
y i=Lici TT0 i2=L
2c i
2T
2
We use the method previously described to calculate the weighted average for thefollowing parameters:
c1
= 0.1 L1=2.0 0.1 y
1=1.80 0.22 T
0=25
c2
= 0.2 L2=2.3 0.1 y
2=1.90 0.41 T = 23 2
Using the error propagation equations, we get for the weighted average:
ytrue
= 1.75 0.19
WEIRD: the weighted average is smaller than either measurement! What's going
on??
cov y1, y 2=c1 c2T2
-
7/30/2019 Error analysis lecture 10
16/32
Physics 509 16
Averaging correlated measurements: example
Bottom line:
Because y1
and y2
disagree, fit
attempts to adjusttemperature tomake them agree.This pushes fittedlength lower.
This is one of manycases in which thedata itself gives
you additionalconstraints on thevalue of asystematic (here,what is the true T).
-
7/30/2019 Error analysis lecture 10
17/32
Physics 509 17
The error propagation equation
Let f(x,y) be a function of two variables, and assume that theuncertainties onxand yare known and small. Then:
f2= dfdx
2
x2 dfdy
2
y22 dfdx
df
dy x yThe assumptions underlying the error propagation equation are:
covariances are known fis an approximately linear function ofxand yover the span ofxdxorydy.
The most common mistake in the world: ignoring the third term.
Intro courses ignore its existence entirely!
-
7/30/2019 Error analysis lecture 10
18/32
Physics 509 18
Example: interpolating a straight line fit
Straight line fit y=mx+b
Reported values from astandard fitting package:
m = 0.658 0.056b = 6.81 2.57
Estimate the value and
uncertainty ofywhenx=45.5:
y=0.658*45.5+6.81=36.75
UGH! NONSENSE!
dy=2.57245.5.0562=3.62
-
7/30/2019 Error analysis lecture 10
19/32
Physics 509 19
Example: straight line fit, done correctly
Here's the correct way to estimate y at x=45.5. First, I find a betterfitter, which reports the actual covariance matrix of the fit:
m = 0.0658 + .056b = 6.81 + 2.57
= -0.9981
dy=2.5720.05645.522 0.99810.05645.52.57=0.16
(Since the uncertainty on each individual data point was 0.5, and thefitting procedure effectively averages out their fluctuations, then weexpect that we could predict the value of y in the meat of thedistribution to better than 0.5.)
Food for thought: if the correlations matter so much, why don't mostfitting programs report them routinely???
-
7/30/2019 Error analysis lecture 10
20/32
Physics 509 20
Reducing correlations in the straight line fit
The strong correlationbetween m and b resultsfrom the long lever arm---since you must extrapolateline to x=0 to determine b, a
big error on m makes a bigerror on b.
You can avoid strongcorrelations by using more
sensible parameterizations:for example, fit data toy=b'+m(x-45.5):
b' = 36.77 0.16
m = 0.658 .085 = 0.43
dy at x=45.5 = 0.16
-
7/30/2019 Error analysis lecture 10
21/32
Physics 509 21
Non-Gaussian errors
The error propagation can give the false impression thatpropagating errors is as simple as plugging in variances andcovariances into the error propagation equation and thencalculating an error on output.
However, a significant issue arises: although the error propagationequation is correct as far as it goes (small errors, linearapproximations, etc), it is often not true that the resultinguncertainty has a Gaussian distribution! Reporting the central
value and an RMS may be misleading.
-
7/30/2019 Error analysis lecture 10
22/32
Physics 509 22
Ratio of two Gaussians I
Consider the ratio R=x/y of two independent variables drawn fromnormal distributions. By the error propagation equation
dRR 2
= dxx 2
dyy 2
Let's suppose x = 1 0.5 and y = 5 1. Then the calculated valueof R = 0.200 .108.
What does the actual distribution for R look like?
-
7/30/2019 Error analysis lecture 10
23/32
Physics 509 23
Ratio of two Gaussians II
x = 1 0.5 and y = 5 1.
Error propagation prediction:R = 0.200 .108.
Mean and RMS of R:0.208 0.118
Gaussian fit to peak:0.202 0.107
Non-Gaussian tails evident,
especially towards larger R,but not too terrible.
-
7/30/2019 Error analysis lecture 10
24/32
Physics 509 24
Ratio of two Gaussians III
x = 5 1 and y = 1 0.5
Error propagation:R = 5.00 2.69.
Mean and RMS of R:6.39 5.67
Gaussian fit to peak:4.83 1.92
Completely non-Gaussian in
all respects. Occasionallywe even divide by zero, orclose to it!
-
7/30/2019 Error analysis lecture 10
25/32
Physics 509 25
Ratio of two Gaussians IV
x = 5 1 and y = 5 1
Error propagation:R = 1 0.28
Mean and RMS of R:1.05 0.33
Gaussian fit to peak:1.01 0.25
More non-Gaussian than first case,much better than second.
Rule of thumb: ratio of two
Gaussians will be approximatelyGaussian if fractional uncertaintyis dominated by numerator, anddenominator cannot be smallcompared to numerator.
-
7/30/2019 Error analysis lecture 10
26/32
Physics 509 26
Ratio of two Gaussians: testing an asymmetry
A word of caution: often scientists like to form asymmetries---forexample, does the measurement from the left half of the apparatusagree with that from the right half? Asymmetry ratios are the usualway to do this, since common errors will cancel in ratio:
A= LR1
2LR
Be extremely careful about using the error on the asymmetry as a
measure of whether A is consistent with zero. In other words,A=0.5 0.25 is usually not a 2 sigma result, probability-wise.
Instead, it's better to simply test whether the numerator is consistent
with zero or not.
G li i h i i
-
7/30/2019 Error analysis lecture 10
27/32
Physics 509 27
Generalizing the error propagation equation
cov fk , fl=i
j
fkx i
flx j
cov x i, x j
If we have N functions of M variables, we can calculate theircovariance by:
We can write this compactly in matrix form as:
G=Gki
=
fk
xi
(an N X M matrix)
Vf=
G
Vx
G
T
A t i
-
7/30/2019 Error analysis lecture 10
28/32
Physics 509 28
Asymmetric errors
The error propagation equation works with covariances. But the
confidence interval interpretation of error bars often reportsasymmetric errors. We can't use the error propagation equation tocombine asymmetric errors. What do we do?
Quite honestly, the typical physicist doesn't have a clue. The mostcommon procedure is to separately add the negative error barsand the positive error bars in quadrature:
Source - Error + Error
Error A -0.025 +0.050
Error B -0.015 +0.010
Error C -0.040 +0.040
Combined -0.049 +0.065
Warning: in spite of how commonthis procedure is and what youmay have heard, it has nostatistical justification and often
gives the wrong answer!
H t h dl t i
-
7/30/2019 Error analysis lecture 10
29/32
Physics 509 29
How to handle asymmetric errors
Best case: you know the likelihood function (at least numerically). Report
it, and include it in your fits.
More typical: all you know are a central value, with +/- errors. Your onlyhope is to come up with some parameterization of the likelihood that
works well. This is a black art, and there is no generally satisfactorysolution.
Barlow recommends (in a paper on his web site):
lnL =12
2
1212
You can verify that this expression evaluates to when =+1
or=2
Parameterizing the asymmetric likelihood
-
7/30/2019 Error analysis lecture 10
30/32
Physics 509 30
a a ete g t e asy et c e oodfunction
lnL =
1
2
2
1212
Note that parameterizationbreaks down when
denominator becomesnegative. It's like sayingthere is a hard limit on atsome point. Use this onlyover its appropriate range ofvalidity.
Barlow recommends this formbecause it worked well for avariety of sample problemshe tried---it's completelyempirical.
See Barlow, Asymmetric
Statistical Errors,arXiv:physics/0406120
Application: adding two measurements with
-
7/30/2019 Error analysis lecture 10
31/32
Physics 509 31
pp gasymmetric errors.
ln L A , B=1
2
A52
1212A5
1
2
B32
1313B3
Suppose A=521
and B=331
. What is the confidence interval for A+B?
Let C=A+B. Rewrite likelihood to be a function of C and one ofthe other variables---eg.
ln L C , A=1
2
A52
1212A5
1
2
CA32
1313CA3
To get the likelihood function for C alone, which is what we care
about, minimize with respect to nuisance parameter A:
lnL C=minA
ln L C , A
Asymmetric error results
-
7/30/2019 Error analysis lecture 10
32/32
Physics 509 32
Asymmetric error results