Proposal for alternative to CLs: Power-Constrained Limits
description
Transcript of Proposal for alternative to CLs: Power-Constrained Limits
![Page 1: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/1.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 1
Proposal for alternative to CLs:Power-Constrained Limits
ATLAS Statistics Forum
CERN, 25 May, 2010
Glen Cowan, RHULKyle Cranmer, NYUEilam Gross, Ofer Vitells, Weizmann Inst.
![Page 2: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/2.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 2
The “CLs” issue
When the cross section for the signal process becomes small (e.g., large Higgs mass), the distribution of the test variable used in a search becomes the same under both the b and s+b hypotheses:
In such a case we will reject the signal hypothesis with aprobability approaching = 1 – CL (i.e. 5%) assuming no signal.
f (q| b)
f (q| s+b)
![Page 3: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/3.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 3
The CLs solutionThe CLs solution (A. Read et al.) is to base the test not onthe usual p-value (CLs+b), but rather to divide this by CLb (one minus the background of the b-only hypothesis, i.e.,
Define:
Reject signal hypothesis if: Reduces “effective” p-value when the two
distributions become close (prevents exclusion if sensitivity is low).
f (q| b)
q
f (q| s+b)
![Page 4: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/4.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 4
Alternative proposal – basic ideaCLs method reduces the p-value according to:
where = strength parameter, proportional to cross section.
Statistics community does not smile upon ratio of p-values;would prefer to regard parameter as excluded if:
(a) p-value of < 0.05(b) power of test of with respect to background-only > some threshold
Requiring (a) alone gives the standard frequentist interval, (CLs+b method) which has the correct coverage.
Requiring ANDed combination of (a) and (b) is more conservative;end effect is similar to CLs, but makes more explicit the minimumthe role of minimum sensitivity (as quantified by power).
![Page 5: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/5.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 5
Similar to….
Feldman and Cousins touched on the same idea in connectionwith FC limits:
We propose to make this more explicit using the power of thetest of a given strength parameter m with respect to thealternative background-only hypothesis.
![Page 6: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/6.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 6
Formalizing the problem
In the context of tests based on the likelihood ratio (),the p-value can be written
The upper limit is found by setting p = and solving for ,
Estimator for strength parameter
Standard normal cumulative dist.
![Page 7: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/7.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 7
False exclusion rate for no sensitivityExcluding if p < gives right coverage, but this meansthat probability to exclude in case of no sensitivity is .
To see this note, probability to exclude assuming = 0 is
“No sensitivity” means / « . In this limit, the false exclusionprobability becomes
![Page 8: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/8.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 8
Power of test of relative to = 0The power of a test of relative to the alternative = 0 is
or equivalently in terms of the distribution of up,
![Page 9: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/9.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 9
Criterion for rejecting We formulate the criterion for rejecting a hypothesized byRequiring p < and also that the power be greater than aminimum threshold 1 – ′. (i.e. Type-II error rate < ′ ). Thepower-constrained limit is thus
where ′ is the for which the power is ′.
The requirement implies sothe minimum power requirement can be expressed
or equivalently
![Page 10: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/10.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 10
Choice of minimum power
Note that if the minimum power 1 – ′ = (typically 0.05), then ′ = 0, and then pc = up always.
Normally would choose < 1 – ′ ≤ 0.5. Convention must bediscussed (also with CMS).
Coverage of power-constrained interval is well defined:
95% for pc = up 100% for pc < up
![Page 11: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/11.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 11
Solution in terms of median p-valueBecause of the monotonic relation between the p-valueand estimator for , the median of pm assuming = 0 is:
In addition to the median (50% quantile) we can also find thequantiles corresponding to the N deviations of muHat:
![Page 12: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/12.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 12
Extra slides
![Page 13: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/13.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 13
A possible experimental outcomeSuppose a given experiment gave the following p-value versus :
Here data have clearly fluctuation low.
![Page 14: Proposal for alternative to CLs: Power-Constrained Limits](https://reader036.fdocuments.us/reader036/viewer/2022062322/56815055550346895dbe54fb/html5/thumbnails/14.jpg)
G. Cowan, RHUL Physics Alternative to CLs page 14
Choice of likelihood ratio statistic
Ongoing discussion as to whether best to use LEP-stylelikleihood ratio
or
and in both cases how to deal with the nuisance parameters.
In simple cases one obtains the same test from both statistics.