Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 1 Upper Limits and Discovery in Search for Exotic...
-
Upload
remington-newson -
Category
Documents
-
view
213 -
download
0
Transcript of Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 1 Upper Limits and Discovery in Search for Exotic...
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 1
Upper Limits and Discovery in Search for Exotic Physics
Jan Conrad
Royal Institute of Technology (KTH)
Stockholm
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 2
Outline
Discovery Confidence Intervals The problem of nuisance parameters
(“systematic uncertainties”)
Averaging Profiling
Analysis optimization
Summary
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 3
General approach to claiming discovery (hypothesis testing)
Assume an alleged physics process characterized by a signal parameter s (flux of WIMPS, Micro Blackholes .... etc.)
One can claim discovery of this process if the observed data is very unlikely to come from the null hypothesis , H0, being defined as non-existence of this process (s=0). ”Very unlikely” is hereby quantified as the ”signifcance” probability αsign, taken to be a small number (often 5 σ ~ 10-7).
Mathematically this is done by comparing the p-value with αsign and reject H0 if p –value < αsign
test statistics, T, could be for example χ2
Actually observed value of the test statistics
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 4
P-values and the Neyman Pearson lemma
Uniformly most powerful test statistic is the likelihood ratio :
For p-values, we need to know the null-distribution of T.
Therefore it comes handy that asymptotically:
Often it is simply assumed that the null-distribution is χ2 but be careful !
see e.g. J.C. , presented at NuFACT06, Irvine, USA, Aug. 2006 L. Demortier, presented at BIRS, Banff, Canada, July 2006
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 5
Type I, type II error and power
Type I error: Reject H0, though it is true.
Prob(Type I error) = α
Type II error: Accept H0, though it is false
Power: 1 - β = 1 – Prob(Type II error)
In words: given H1, what is the probability that we will reject H0 at given
significance α ? In other words: what is the probability that we detect H1 ?
In designing a test, you want correct Type I error rate (this controls the number of false detections) and as large power as possible .
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 6
Why 5 ?
… traditional: we have seen 3 significances disappear (….we also
have seen 5 signficances disappear on the other hand ….)
Principal reasoning (here done for the LHC):
LHC searches: 500 searches each of which has 100 resolution elements (mass, angle bins, etc.) 5 x 104 chances to find something.
One experiment: False positive rate at 5 (5 x 104) (3 x 10-7) = 0.015. OK !
Two experiments: Assume we want to produce < 100 unneccessary theory papers allowable false positive rate: 10. 2 (5 x 104) (1 x 10-4) = 10 3.7 required. Required other experiment verification:
(1 x 10-3)(10) = 0.01 3.1 required.
It seems that the same reasoning would lead to smaller required signficance probabilities for EP searches in NT.
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 7
Confidence Intervals (CI)
Instead of doing a hypothesis test, we might want to do a interval estimate on the parameter s with confidence level 100(1 – α) % (e.g. 90 %):
Bayesian:
Frequentist:
Invert by e.g. Neyman construction of confidence intervals (no time to explain)
- special case 1: n 2= upper limit - special case 2: two sided/one sided limits depending on observation Feldman & Cousins
Confidence intervals are often used for hypothesis testing.
G. Feldman & R. Cousins, Phys. Rev D57:3873-3889
See e.g. J.C. presented at NuFACT06, Irvine, USA, Aug. 2006
K. S. Cranmer, PhyStat 2005, Oxford, Sept. 2005
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 8
Nuisance parameters1)
Nuisance parameters are parameters which enter the data model, but which are not of prime interest (expected background, estimated signal/background efficiencies etc. pp., often called systematic uncertainties)
You don’t want to give CIs (or p-values) dependent on nuisance parameters need a way to get rid of them
1) Applies to both confidence intervals and nuisance parameters
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 9
How to ”get rid” of the nuisance parameters ?
There is a wealth of approaches to dealing with nuisance parameters. Two are particularly common:
Averaging (either the likelihood or the PDF):
Profiling (either the likelihood or the PDF):
... less common, but correct per construction: fully frequentist, see e.g:
Bayesian
G. Punzi, PHYSTAT 2005, Oxford, Sept. 2005
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 10
NT searches for EP: why things are bad ..... and good.
Bad Low statistics makes the use of asymptotic methods
doubtful systematic uncertainties are large.
Good: Many NT analyses are single channel searches with
relatively few nuisance parameters
rigorous methods are computationally feasible (even fully frequentist)
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 11
Coverage
A method is said to have coverage (1-α) if, in infinitely many repeated experiments the resulting CIs include (cover) the true value in a fraction (1-α) of all cases (irrespective of what the true value is).
Coverage is a necessary and sufficient condition for a valid CI calculation method
1 -α
s
1
0.9over-covering
under-covering
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 12
Averaging: hybrid Bayesian confidence intervals
J.C, O. Botner, A. Hallgren, C. de los Heros Phys. Rev D67:012002,2003 R. Cousins & V. Highland Nucl. Inst. Meth. A320:331-335,1992
Example PDF:
Perform Neyman-Construction with this new PDF (we will assume Feldman & Cousins in the remainder of this talk)
Treats nuisance parameters Bayesian, but performs a frequentist construction.
Integral is performed in true variables Bayesian
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 13
Coverage of hybrid method.
J.C & F. Tegenfeldt , PhyStat 05, Oxford, Sept. 2005, physics/0511055
(1-
α) M
C
true s true sF.Tegenfeldt & J.C. Nucl. Instr. Meth.A539:407-413, 2005
Use Log-normal if large uncertainties !!!!!
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 14
Commercial break: pole++
Bayesian treatment in FC ordering Neyman construction treats P(n|εs +b)
Consists of C++ classes: Pole calculate limits Coverage coverage studies Combine combine experiments
Nuisance parameters supports flat, log-normal and Gaussian uncertainties in efficiency and
background Correlations (multi-variate distributions and uncorrelated case)
Code and documentation available from: http://cern.ch/tegen/statistics.html
J.C & F. Tegenfeldt , Proceedings PhyStat 05, physics/0511055
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 15
Example: hybrid Bayesian in NTs
From Daan Huberts talk (this conference):
with systematicswithout systematics
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 16
Profiling: Profile Likelihood confidence intervals
Lower limit Upper Limit
2.706
meas n, meas. b MLE of b given s
MLE of b and s given observations
To extract limits:
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 17
From MINUIT manual
See F. James, MINUIT Reference Manual, CERN Library Long Write-up D506, p.5:
“The MINOS error for a given parameter is defined as the change in the value of the parameter that causes the F’ to increase by the amount UP, where F’ is the minimum w.r.t to all other free parameters”.
Confidence Interval
Profile Likelihood ΔΧ2 = 2.71 (90%),
ΔΧ2 = 1.07 (70 %)
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 18
Coverage of profile likelihood
Background: Poisson (unc ~ 20 % -- 40 %) , Efficiency: binomial (unc ~ 12%) Rolke
et alMinuit
W. Rolke, A. Lopez, J.C. Nucl. Inst.Meth A 551 (2005) 493-503
(1-
α) M
C
true s
Available as TRolke in ROOT !
Should be able to treat common NT cases
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 19
Profile likelihood goes LHC.
Basic idea: calculate 5 σ confidence interval and claim discovery if s = 0 is not included.
Straw-man model:
Typical: b = 100, т = 1 ( 10 % sys. Uncertainty on b)
K. S. Cranmer, PHYSTAT 2005, Oxford, Sept. 2005
Method Calculated coverage (nom. 5σ)
Critical number of events (bobs = 100)
Ignore ”systematics” 3.1 150
Bayesian treatment 4.2 178
Profile likelihood 5.0 185
Size of side band region
- 17 events!!
- 35 events!!
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 20
Analysis optimisation
Consider some cut-value t. Analysis is optimised defining a figure of merit (FOM). Very common:
Alternatively, optimize for most stringent upper limit. The corresponding figure of merit is the model rejection factor, MRF:
G. Hill & K. Rawlins, Astropart. Phys. 19:393-402,2003
Mean upper limit (only bg)
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 21
In case of systematics ?
Simplest generalizations one could think of:
In general, I do not think it makes a difference unless:
NO !
Yes !
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 22
Optimisation for discovery and upper limit at the same time ?
Fix significance (e.g αsign = 5 σ) and confidence level (e.g. 1-αCL = 99 %). Then define sensitivity region in s by :
The FOM can be defined to optimize this quantity (e.g simple counting experiment):
G. Punzi, PHYSTAT 2003, SLAC, Aug. 2003
Signal efficiency
Number of σ (here assumed αsign = 1 – αCL)
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 23
Conclusions/Final Remarks
Two methods to calculate CI and claim discovery in presence of ”systematic” uncertainties have been discussed.
The methods presented here are certainly suitable for searches for Exotic Physics with Neutrino Telescopes and code exists which works ”out of the box”
Remark: the ”simplicity” of the problem (single channel, small number of nuisance parameters) make even rigorous methods applicable
Remark 2: the LHC example shows that for large signficances (discovery) hybrid Bayesian might be problematic.
I discussed briefly the issue of sensitivity and analysis optimisation.
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 24
Backup Slides
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 25
B0s µ+µ-
CDF1 CDF 2
Eff uncertainty [%] 18.2 16.0
Eff uncertainty [%] 20.3 19.2
Corr. eff. Uncertainty [%]
15.5
95 % CI [10-7] 2.5 4.3
95 % combined limit [10-7]
1.7 (2.0 in Bernhard et. al.)
J.C & F. Tegenfeldt , Proceedings PhyStat 05, physics/0511055
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 26
Neyman construction
J. Neyman, Phil. Trans. Roy. Soc. London A, 333, (1937)
Exp 1Exp 2
Exp 3
One additional degree of freedom: ORDER in which you inlcude the n into the belt
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 27
Average coverage
s
Max/Min coverage
Ordering function: (Punzi, PhyStat05)
Can be any ordering in prime observable sub-space, in this case
Likelihood ratio (Feldman & Cousins)
~ FC Profile
s
Poisson signal, Gauss eff. Unc (10 %)
Projection method with appropriate ordering.
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 28
FC ordering: coverage
(1-
α) M
C
true s
Nominal coverage
Calculated by Pseudo-experiments
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 29
Some methods for p-value calculation
Conditioning Prior-predictive Posterior-predictive Plug-In Likelihood Ratio Confidence Interval Generalized frequentist
Jan Conrad (KTH, Sthlm) EPNT 06 20 September 2006 30
Some methods for confidence interval calculation (the Banff list)
Bayesian Feldman & Cousins with Bayesian treatment of nuisance
parameters (Hybrid Bayesian) Profile Likelihood Modified Likelihood Feldman & Cousins with Profile Likelihood Fully frequentist Empirical Bayes