Results of combination Higgs toy combination, within and across experiments, with RooStats

Results of combination Higgs toy Results of combination Higgs toy combination, within and across combination, within and across

experiments, with RooStatsexperiments, with RooStats

GrGrégory Schottégory Schott

Institute for Experimental Nuclear Physicsof the Karlsruhe Institute of Technology

on behalf of the ATLAS-CMS Higgs groups and statistics forum

ATLAS-CMS statistics meeting, 5th July 2010

2Grégory Schott - ATLAS-CMS statistics meeting - 01.07.2010

Introduction I will present the results of ATLAS, CMS and the combination

summary of the results obtained by Kyle Cranmer and myself thanks to Kyle for the FC, PL and some Hybrid results in case of PL we both get results that of course agree very well

This is on-going work; while a large part of the results are available, not everything has been done yet and not everything has been validated yet

Toy study (made with mockup numbers), H→WW at 160 GeV with 1fb-1

don't draw physics conclusion of the results

Significance and upper limit computed with RooStats (for a given data) different approaches tested as recommanded by the statistics forums

Validation of RooStats (ROOT 5.27.04, June 2009) Conclusion and discussions


The combined model Combination of H→WW→ee, , e analyses Parameter of interest: ratio of cross-sections, r = = / SM

CMS analysis: 3 observables, 37 nuisance parameters Lognormal distribution of nuisance parameters Uncertainty on control region measurement in the systematics

ATLAS analysis: 9 observables, 12 nuisance parameters Truncated gaussian distribution of nuisance parameters Control regions measurements included in the analysis

Combined analysis: common parameter interest (thus, 100% correlated) no other correlation (yet) across the experiments

Concretely, all the information for the combination is shared via 2 ROOT files (for ATLAS & CMS) containing RooFit/RooStats workspaces

graphical representation of the objects in the combined

model


Toy-data Data used for significance:

generated from the model in the hypothesis that signal is present

Data used for exclusion: generated from the model in the background-only hypothesis

Check the backups if you're interested by the number of observed events assumed (or the likelihood functions)


Profiled likelihood results

significance: reasonably good estimate if large enough

95% CL UL taken for -log (r) = 1.921 assumption of Wilks's asymptotia that's not always valid!

with systematics

r UL

ATLAS 0 0.79CMS 0 0.28

COMBI 0 0.25

with systematics

r signif.

ATLAS 1.13 2.70CMS 0.98 4.89

COMBI 1.02 5.58

^

^


Hybrid Frequentist-Bayesian

run for the 3 test statisticsmentionned in the previoustalk

need to study the advantageand inconvenient of the 3approaches

test statistics significance (no syst.)

significance (with syst.)

ATLASQLEP

QTEV

()

3.78-

3.07 ± 0.012.8 ± 0.1

-

CMSQLEP

QTEV

()

6.22 ± 0.02--

4.77 ± 0.02> 4.6

4.3 ± 0.1

COMBIQLEP

QTEV

()

---

> 4.6> 3.5

-

CMS, 477k toys

QLEP()QTEV

B-toysSB-toys

data


CMS model with 'LandS'

Hybrid test statistics distributions

computing the p-value for significancein this approach is challenging:

speed improvements would be useful or use importance sampling techniques

CMS distribution (and results previousslide) made with a RooFit-independenttool

QLEPQLEP


Summary of upper limits

technique test stat rule sampling UL ATLAS UL CMS UL COMBIFeldman-Cousins (no

syst.)() CLS+B

toys 0.69 ± 0.05 - -

Profile LR (Wilks) () CLS+Basymptotic 0.79 0.28 0.25

Feldman-Cousins++ () CLS+Btoys 0.78 ± 0.05 0.26 ± 0.02 0.23 ± 0.02

Hybrid QLEP CLStoys ~ 0.68 0.29 ± 0.03

(LandS)-

Hybrid QLEP CLS+Btoys ~ 0.61 - -

Bayesian n/a, flat prior on r MCMC* 0.72 0.31 0.28

ATLAS

r

post

erio

r (r |

dat

a)

CMS

with BAT [ Caldwell, Kollar, Kroeninger, Comp.Phys.Com. 180, 2197 (2009) ]

95% CL UL

95% CL upper limits: results with systematics (except if indicated otherwise)


Validations Some validation of the analysis performed with independent software

(LandS, M. Chen, CMS) Since 'LandS' was developped specifically for this analysis, it is

faster than RooStats which is a general tool showing that RooStats still need to improve

table of compared results: we still need to work on the missing parts

PL and toy-MC results using an earlier version of the ATLAS model were validated with an independent code (Hao Liu, et. al.)

further on-going work on validating Bayesian limits: S. Schmitz (CMS) currently using BAT, RooStats providing another MCMCCalculator


Conclusion and outlook A good start, work is on-going on

Lessons learned: we achieved in a short time scale to combine the Atlas and

CMS analyses -> proof of principle that we can do that proof of principle we can compare different statistical methods

to one another still some weaknesses identified that RooStats need to improve we also need to continue the validations of RooStats

still need to understand those results discussion has started on statistical methods we want to use in common

proper treatment of correlations across experiments will require planning and thoughtful parametrization of systematics

activity will continue (other masses, other channels, more thorough correlation studies, ...)


End of talk−

Backup slides


Toy-data Data used for significance:

generated from the model in the hypothesis that signal is present

Data used for exclusion: generated from the model in the background-only hypothesis Observables:

1) nobs_bin1 = 4 2) nobs_bin2 = 2 3) nobs_bin3 = 3 4) obs_s_em_0j = 19 5) obs_ww_em_0j = 63 6) obs_tt_em_0j = 3814 7) obs_wj_em_0j = 123 8) obs_s_ee_0j = 6 9) obs_ww_ee_0j = 14 10) obs_wj_ee_0j = 53 11) obs_s_mm_0j = 16 12) obs_ww_mm_0j = 26

Observables: 1) nobs_bin1 = 15 2) nobs_bin2 = 7 3) nobs_bin3 = 13 4) obs_s_em_0j = 36 5) obs_ww_em_0j = 52 6) obs_tt_em_0j = 3650 7) obs_wj_em_0j = 143 8) obs_s_ee_0j = 9 9) obs_ww_ee_0j = 5 10) obs_wj_ee_0j = 49 11) obs_s_mm_0j = 18 12) obs_ww_mm_0j = 28


slide from K. Cranmer, 01.07.2010

ATLAS likelihood model


CMS likelihood model

slide from A. Korytov, 24.06.2010


Profile likelihood (zoom)


Bayesian upper limits

r

posterior (r | data)ATLAS

posterior (r | data)COMBI

r

Results of combination Higgs toy combination, within and across experiments, with RooStats

Documents

Transcript of Results of combination Higgs toy combination, within and across experiments, with RooStats