880.P20 Winter 2006 Richard Kass 1 Confidence Intervals and Upper Limits Confidence intervals (CI)...

880.P20 Winter 2006 Richard Kass 1

Confidence Intervals and Upper LimitsConfidence intervals (CI) are related to confidence limits (CL). To calculate a CI we assume a CL and find the values of the parameters that give us the CL.

Caution

CI’s are not always uniquely defined.

We usually seek the minimum interval or symmetric interval.

Example: Assume we have a gaussian pdf with =3 and =1. What is the 68% CI ?We need to solve the following equation:

ba dxxG )1,3,(68.0

Here G(x,3,1) is the gaussian pdf with =3 and =1. There are infinitely many solutions to the above equation. We (usually) seek the solution that is symmetric about the mean ():

cc dxxG

)1,3,(68.0To solve this problem we either need a probability table, or remember that 68% of the area of a gaussian is within of the mean.Thus for this problem the 68% CI interval is: [2,4]Example: Assume we have a gaussian pdf with =3 and =1.

What is the one sided upper 90% CI ?Now we want to find the c that satisfies:

Using a table of gaussian probabilities we find 90% of the area in the interval [-, +1.28]Thus for this problem the 90% CI is: [-, 4.28]

c dxxG )1,3,(9.0

need to solve for a and b.


Poisson Upper LimitsSuppose an experiment is looking for the X particle but observes no candidate events.

What can we say about the average number of X particles expected to have been produced?

First, we need to pick a pd (or pdf). Since events are discrete we need a discrete pd Poisson.Next, how unlucky do you want to be ? It is common to pick 10% of the time to be unlucky.

We can now re-state the question as:“Suppose an experiment finds zero candidate events. What is the 90% CL upper limit on theaverage number of events () expected assuming a Poisson probability distribution ?”

1 !9.0

n

n

n

eCL

We need to solve for in the following equation:

In practice it is much easier to solve for 1-CL:

)1ln(!!

1101

CLen

e

n

eCL

n

n

n

n

For our example, CL=0.9 and therefore =2.3 events.

So, if =2.3 then 10% of the time we should expect to find 0 candidates. There was nothing wrong with our experiment. We were just unlucky.

Example: A cosmic ray experiment with effective area=103 km2 looks for events with energies >1020 eV and after one year has no candidate events. We can calculate a 90% UL on the flux of these high energy events: Flux< 2.3x10-3/km2/year @ 90% CL.


Poisson Upper LimitsExample: Suppose an experiment finds one candidate event. What is the 95% CL upper limit on the average number of events () ?

74.4!!

111

02

een

e

n

eCL

n

n

n

n

Here we are saying that we would get 2 or more events 95% of the time if =4.74.

The 5% includes 1 AND 0 events.

2004 PDG has a good table (32.3, P286) for these types of problems. Things get much more interesting when we have background in our data sample! We measure N events & we predict B background events The number of our signal events, S, is: S=N-BThree interesting situations can arise: I) How should we handle the case where B Usually B is calculated without knowledge of the value of N. Since B and N are obtained independently there is nothing that guarantees that N>B

II) Even if N > B a sloppy background prediction can lead to a better (smaller) UL than a careful background prediction! For fixed N, larger B means smaller value of S smaller UL.

III) More background is better than less background?Consider 2 experiments looking for high energy cosmic ray events. Both experiments have 6 candidates.Exp1 estimates 2 background events, Exp2 estimates 4 background events. Exp2 will have the lower UL!

Good discussion in PDG on how to handle these situations….


Exponential MLM Example Again

-67

-66

-65

-64

-63

-62

0 100 200 300 400 500 600

lnL

lnL

-5.613 104

-5.613 104

-5.613 104

-5.613 104

-5.613 104

-5.613 104

97 98 99 100 101 102 103 104

lnL

lnL

y = m3-(m0-m1)^ 2/(2*m2^ 2)

ErrorValue0.013475100.8m1

0.00889441.01m2

0.034297-56128m3 NA0.055864Chisq

NA0.99862R

Example: Exponential decay: /),(: /tetfpdf

Log-likelihood function for 10 eventsLnL max for =1891 points: (140, 265)L not gaussian

Generate events according to an exponential distribution with = 100Calculate lnL vs (time) andfind maximum of lnL and the points where lnL =linLmax-1/2 (“1 points”)

n

i

n

ii

t tnLandeL i

1 1

/ /lnln/

Log-likelihood function for 104 eventsLnL max for =100.81 points: (99.8, 101.8)L is fit by a gaussian

Suppose we want to calculate a CI but can’t simply invert the pd or pdf….Can (usually) do the CI calculation with a Monte Carlo Simulation

A simple MC likethis is often called a “Toy Monte Carlo”


Maximum Likelihood Method Example

How do we calculate confidence intervals for our MLM example?

For the case of 104 events we can just use gaussian stats since the likelihoodfunction is to a very good approximation gaussian. Thus the “1 points” will giveus 68% of the area under gaussian curve, the “2 points” points ~95% of area, etc.

Unfortunately, the likelihood function for the 10 event case is NOT approximated by a gaussian. So the “1 points” do not necessarily give you68% of the area under the gaussian curve.

In this case we can calculate a confidence interval about the mean using a MonteCarlo calculation as follows:1) Generate a large number (e.g. 107) of 10 event samples each sample having a mean

lifetime equal to our original 10 event sample (*=189)2) For each 10 event sample calculate the maximum of the log-likelihood function (=i)3) Make a histogram of the i’s. This histogram is the pdf for . 4) To calculate a X% confidence interval about the mean, find the region where

X%/2 of the area is in the region [L, *] and X% is in the region [*, H]. NOTE: since the pdf may not be symmetric around its mean, we may not be able to find equal area regions below and above the mean.


Maximum Likelihood Method Example

0

1 105

2 105

3 105

4 105

5 105

6 105

7 105

0 100 200 300 400 500

even

ts/1

0

1

10

100

1000

104

105

106

0 100 200 300 400 500

even

ts/1

0

Semi-logLinear

Above is the histogram or pdf of 107 ten event samples each with *=189. By counting events (i.e. integrating) in an interval around *, the histogram (actually, I printed out the number of events in one unit steps from 0 to 650) gives the following:

**

“±1 region”: 34% of area in regions (139189) and (189263)90% CI region: 45% of area in regions (117189)] and (189421)

54.9% of the area is in the region (0189)

The upper 95% region (i.e. 47.5% of the area above the mean) is not defined.

Very closeTo likelihood result

NOTE: the variance of an exponential distribution can be calculated analytically:ndtet t //)( 2/

0

22

Thus for the 10 event sample we expect = 60, not too far off from the 68% CI!For the 104 event sample, the CI’s from the ML estimate of and the analytic are essentially identical (both give =1.01).


Confidence Regions & MLMOften we have a problem that involves two or more parameters. In these instancesit makes sense to define confidence regions rather than an interval. Consider the case where we are doing a MLM fit to two variables , .

Previously we have seen that for large samples the Likelihood function becomes “gaussian”:

Consider the case of two correlated variables , :

withLL))((

2)()(

)1(

1

2

1ln),(ln

**

2

2*

2

2*

2max

2

2*

max

)(

2

1ln)(ln

LL

))((

2)()(

)1(

1 **

2

2*

2

2*

2Q

The contours of constant likelihood are given by:Q

eLLQLL 2

1

maxmax ),(2

1ln),(ln

We can re-write Q in matrix form, with V-1 the inverse of the error matrix:

1

*

*

2

2

*

*

2 /1/

//1

)1(

1

VQ T

T

We can generalize this to n parameters with V the nxn error matrix and an n-dimensional vector


Confidence Regions & MLMThe variable Q is described by a 2 pdf with #dof= # of parameters For the case where the parameters have gaussian pdfs this is exact For the non-gaussian case this is true in the limit of a large data sampleNote: we can re-write Q in the expected form of a 2 variable by transforming from correlated to uncorrelated variables. For the 2D case the transformation is a rotation:

2

2*

2

2***

2

2*

2

2* )()())((2

)()(

yx

yyxx

22*

*

*

* 22tanwith

cossin

sincos

yy

xx

Since we know the pdf for Q we can calculate a confidence level for a fixed value of Q.The case of 2 variables is easy since the 2 pdf is just:

Q0 CL(%)1 392.3 684.6 6.2 959.2 99

00

2

1

0

2

1

0 121)(

QQ

QedQeQQPCL

2/22/12/22/

2

2

1)2,(][

)2/(2

1),(

2 Qnn

epen

np

We can calculate the confidence level for the region bounded by a fixed value of Q:


Confidence Regions Example

0 10 20 30 40

0

10

20

30

40

Example: Suppose the BaBar experiment did a maximum likelihood analysis to searchfor B and B events. The results of the MLM fit are:

N =164, NK =255, =0.5 (warning: these are made up numbers!)

N and NK are highly correlated, since at high momentum (>2GeV) BaBar has a hard time separating ’s and K’s.

NK

N

45

)16)(25()5.0(2

4

)16(

5

)25(

)5.1(

12

2

2

2

2

NNNNQ KK

The contours of constant probability are given by:

99%, Q=9.295%, Q=6.2

68%,Q=2.3

39%, Q=1

ab

Point “a” is excluded at the 95%CLPoint “b” is excluded at the 99%CL


Confidence Regions Examples From PDG 2004

mass of Higgs Vs mass of top quark

solar neutrino oscillations experiments

Both examples show allowed regions at various confidence levels.

880.P20 Winter 2006 Richard Kass 1 Confidence Intervals and Upper Limits Confidence intervals (CI)...

Documents

Transcript of 880.P20 Winter 2006 Richard Kass 1 Confidence Intervals and Upper Limits Confidence intervals (CI)...