News from the Statistics Forum

G. Cowan News from the Statistics Forum / 2 February, 2012 1

News from the Statistics Forum

ATLAS Week Plenary MeetingCERN, 2 February, 2012

Glen Cowan (RHUL)

Alex Read (Oslo)


Overview

Summarize recent progress (meetings 8.11.11, 19.12.11, 9.1.12, plus very active email/hypernews discussions).

Next Statistics Forum meeting is today (2.2.12), 17:00, Salle Dirac

Main topics for today:

Refinement for frequentist tests (“uncapped” p-values)

Progress on Bayesian reference analyses

ABCD method revisited


Refinement for Frequentist Tests

Aaron Armbruster, Tim Ayde

The usual statistic we define to test, e.g., the background-onlyhypothesis reflects disagreement with the hypothesis only ifthe data fluctuate higher than the expected background:

This means that the p-value of the background-only hypothesis,under assumption of background only, is 0.5 ~half the time.

Bug or feature? The interesting cases are when the p-value issmall, e.g., p0 < 2.9 × 107 means we reject background-only hypothesis at 5σ level.

But the plot of p0 versus mH comes out “capped” at 50%.


Capped p-valuesHere data lower than expected background, p0 capped at 0.5



Similar capping for CLs



Redefinition of test statistic

Replace q0 by



Uncapped p-values



Summary on (un)capped p-values

By modifying the test statistic as proposed, there is no change in cases where the p-value is less then 0.5, so no important conclusions change.

But if the data fluctuate away from discovery or exclusion,the new statistic makes this easily visible.

Some technical issues concerning asymptotic properties of q0′ still under study, but providing these are resolved we will propose to ATLAS (and CMS) that the “uncapped” method be adopted as part of the standard frequentist procedure.



Progress on Bayesian methods

Although main emphasis has been on frequentistmethods, also important progress on Bayesian techniques:

E.g., many contributions to Statistics Forum on Bayesian Analysis Toolkit (BAT) and Bayesian tools within RooStats.


Bayesian priors

In the statistics community there is wide use of so-called “reference”priors. Recently Diego has developed a method to construct the reference prior for the signal parameter s in counting experiments:

D. Casadei, Reference analysis of the signal + background model in counting experiments, JINST 7 (2012) P01012; arXiv:1108.4270

For various reasons (historical,simplicity, conservatism,...) the prior for the mean number of signal events, s, is usually taken as constant:

This has many well-known features: very conservativefor limits, not invariant under reparametrization,...


Reference analysis à la Diego

Find marginal model for the number of events, k, given a mean signal s and background b, by assuming b has a gamma prior:

Use this model to compute the Fisher information:

And then use this to find the reference prior for s: (same as theJeffreys prior for single parameter):

For details see Diego’s paper!


Example of reference priors π(s) for different

gamma priors used for the backgroundD. Casadei, JINST 7 (2012) P01012; arXiv:1108.4270


Status of Bayesian Recommendation

The current ATLAS recommendation on limits is to use (frequentist) CLs, except for analyses with a history of using Bayesian limits (and if CMS also follows this), in which case a constant prior is used for the signal rate.

The reference prior has a number of important advantages:

fast convergence of limit’s coverage probability tothe (Bayesian) probability content;

invariance under reparametrization,...

We would like to encourage those people doing Bayesian analyses to try out the reference prior so that we can gain experience with it.


Discussions on ABCD MethodAlex, Eilam, Ohad,...

Basic idea:

Measure x and y(~uncorrelated for bkg)

Signal only in A;estimate backgroundin A by:

Refinements for correlations, spillover of signal into B, C, D,...

Difficulty arises when using error propagation for error of background estimate if some of the boxes have few events.


“Write down the likelihood”

Treat the numbers of events found in A, B, C, D as independentand Poisson distributed; the likelihood is:

For details see note by Alex on twiki (Statistics Tools in ATLAS):https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StatisticsTools

Use the likelihood to test different values of the signal rate(e.g., profile likelihood ratio); or use in Bayesian analysis.

Straightforward generalization to include further refinements(correlations, crossover, systematics, etc.)

Alex, Eilam, Ohad,...


Some other recent and ongoing discussionsUnfolding

Main effort in SM, exotics, top – B. Malaescu, F. Spanohttps://indico.cern.ch/conferenceDisplay.py?confId=157136https://indico.cern.ch/conferenceDisplay.py?confId=156510

Plotting differences between data and expectation

D. Casadei and G. Choudalakis, arXiv:1111.2062

Issues related to unexpectedly tight constraints on nuisanceparameters from profile-likelihood method.

Revisiting the “tevatron” likelihood ratio q = 2 ln (Ls+b/Lb) (GC).

Multiple regression (Samir Ferag), SusyFitter (Dan Short)

Asymptotic combination of uncorrelated channels (Ohad Silbert)

Your input is welcome!


Extra slides


Unfolding twiki page (SM & Top Groups)

https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/StandardModelUnfolding


Workshop on how to package our results


Reminder of Editorial Board Guidelines

Ensure the analysis follows the recommendations of the ATLAS Statistics Forum with regard to statistical procedures (e.g. limit setting, unfolding, combinations of data). In cases of doubt, or when complex or unusual procedures are being used, make sure the authors discuss with the Statistics Forum conveners before the analysis reaches the approval stage.

https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/EditorialBoardGuidelines

News from the Statistics Forum

Documents

Transcript of News from the Statistics Forum