BanffChallenge2

TomJunkFermilab

BIRSSta6s6csinHEPWorkshopJuly2010

1TomJunkBanffChallenge2

TomJunkBanffChallenge2 2

CommonStandardsofEvidence

Physicistsliketotalkabouthowmany“sigma”aresultcorrespondstoandgenerallyhavelessfeelforp‐values.

Thenumberof“sigma”iscalleda“z‐value”andisjustatransla6onofap‐valueusingtheintegralofonetailofaGaussian

Double_tzvalue=‐TMath::NormQuan6le(Double_tpvalue)

1σ⇒15.9%

Tip:mostphysiciststalkaboutp‐valuesnowbuthardlyusethetermz‐value

Folklore:95%CL‐‐goodforexclusion3σ:“evidence”5σ:“observa6on”Someargueforamoresubjec6vescale.

€

pvalue =1− erf zvalue / 2( )( )

2z-value (σ) p-value

1.0 0.159

2.0 0.0228

3.0 0.00135

4.0 3.17E-5

5.0 2.87E-7

BanffChallenge2Problem#1–StackedplotshownHEP‐style

3TomJunkBanffChallenge2

• Observeddatashownaspointswithsqrt(n)errorbars(yes,theconven6on’scrazybutthat’sthewaywedoit.)• Signalpredic6onshownstackedontopofthebackgroundpredic6on.UsefulbecausewecancomparethethedatawithH0andH1withjustoneplot.

DiscriminantVariable

Even

ts

nbackground=10000nsignal=210ndata=9815

TomJunkBanffChallenge2 4

€

−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )

L(data |b, ˆ ˆ θ )

Problem1,nosystema6cuncertainty1MillionsimulatedexperimentsforH0and1MillionsimulatedexperimentsforH1

Nuisanceparametersalwaysattheirnominalvalues

p‐value=5.95x10‐4z‐value=3.24‐2lnQobs=1.98

hatsdon’tmaierheresincethere’snofit.

TomJunkBanffChallenge2 5

€

−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )

L(data |b, ˆ ˆ θ )

Problem1,withsystema6cuncertainty1MillionsimulatedexperimentsforH0and1MillionsimulatedexperimentsforH1

p‐value=1.91x10‐5z‐value=4.11‐2lnQobs=‐15.43

nowdotwofitspersimulatedexperiment‐‐fitforallnuisanceparameters,rateandshape

Eachpseudoexperimentgetsrandomlyfluctuatednuisanceparameters(“prior‐predic6veensemble”)

TomJunkBanffChallenge2 6

ATricktoUseonly1MSimulatedH0Experiments

• Fitthedistribu6onof‐2lnQ(H0)toasumoftwoGaussians–canintegratethatanaly6callywitherf’s.• Needtocheckfitquality.Arealjobwouldbetoes6matetheuncertainty(extrapola6onuncertaintyifneedbe).• Forarealdiscoveryofapar6cle,we’djustusetheneededCPU.Maybethefiiergetsstuckoncein1x107experiments–needtoknowthat.

ThesumoftwoGaussiansisagoodapproxima6onherebutapooroneiftheproblemismorediscrete–onebin,forexample,orlotsoflows/bbinsandoneveryhighs/bbinwithjustafewexpectedeventsinit.

TomJunkBanffChallenge2 7

AProblemwithProblem#2

DiscriminantVariable

Even

ts

nbackground=10000nsignal=200ndata=98439209dataeventsinthevisiblepartofthehistogram

634observedeventsoutof9843areintheupperoverflowbin(!)(~6.44%ofthem)Backgroundtemplatemodelsthis.

IdiscouragetheuseofROOT’sover‐andunderflowbinsforseveralreasons:

1)Theyarenot(usually)ploied.Hardtovalidatethemifyoucannotseethem2)TheyarenotincludedinTH1::Integral()orinfSumwwhendumped.Soscalingbydividingbytheintegralandmul6plyingbythedesiredyieldwon’tgetitright.

rootaccumulatesentriesbeyondthehistogramedgesinunderflowandoverflowbins,andtreatsthemasspecialbins(why?)Sugges6ontoallstudents:constrainallselecteddatatobeinvisiblebins(maxandmin). Problems1and3havenoentriesinthe

underfloworoverflowbins.

SoIsolvedaproblemthatisslightlydifferentandpossiblymoreinstruc6ve.

TomJunkBanffChallenge2 8

Problem2

€

−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )

L(data |b, ˆ ˆ θ )

Nosystema6cs:‐2lnQ=17.46z‐value=0.20p‐value=0.42

Withsystema6cs:‐2lnQ=‐17.33z‐value=4.1p‐value=1.75x10‐5

DiscriminantVariable

Even

ts

GOFnotevaluatedwithoutsystema6cs–preiypoorthough.Showsthattheno‐systema6csinterpreta6onisincorrect.

nbackground=10000nsignal=200ndata=9843

TomJunkBanffChallenge2 9

Problem2’sFit

Notperfectonthetail,probablyjustneedtorunmorepseudoexperiments

TomJunkBanffChallenge2 10TomJunkBanffChallenge2 10

Problem3

€

−2lnQ ≡ LLR ≡ −2ln L(data | s+ b, ˆ θ )

L(data |b, ˆ ˆ θ )

Nosystema6cs:‐2lnQ=‐43.1z‐value=7.3p‐value=1.4x10‐13

Withsystema6cs:‐2lnQ=‐21.2z‐value=4.44p‐value=4.5x10‐6

DiscriminantVariable

Even

ts

nbackground=80nsignal=72ndata=134

TomJunkBanffChallenge2 11

CombiningProblems1+2+3Jointfits–correlatedsystema6cuncertain6esinarealproblem.Wearetoldtodecorrelatethenuisanceparametersbetweenchannels.

Nosystema6cs:

‐2lnQcomb=‐2lnQ1‐2lnQ2‐2lnQ3

Withsystema6cs–spoiledabitbythedifferentfits,ifnuisanceparametersarecorrelated.Inthiscasethesumrules6llworksbecausealldataandallnuisanceparametersareindependent.

GOFispoorforbothhypotheses–seeprob.2.Largesensi6vity.Nosystema6cscanruleoutbothH0andH1.

Nosystema6cs:‐2lnQ=‐23.7z‐value=7.0p‐value=1.2x10‐12

Withsystema6cs:‐2lnQ=‐53.96z‐value=7.5p‐value=3.6x10‐14

TomJunkBanffChallenge2 12

Combina6onSignificancesareaBitofanExtrapola6onwithjust1MSimulatedOutcomes

TomJunkBanffChallenge2 13

Es6matesofSensi6vity• Well,1Millionsimulatedexperimentsisn’tenough–canget1Millionmoreincombina6onbyaddingthe‐2lnQ’sfrom1,2,and3’stogether.

• Wilks’sTheoremprobablyisagoodapproxima6onheretoo.

• Importancesamplingcouldbeusedtoimproveprecisionintails

• Fordiscovery,we’duserealCPUasthesysteam6cswillbecorrelatedandtheremaybeasinglebinaddingadiscretecomponenttoit.

• Ourfavoritesensi6vityes6mate:pmed,signalisthemedianexpectedp‐valueassumingasignalispresent.1Mpseudoexperimentsnotquiteenough.

• Astand‐in:the“o‐value”(namedbytheCDFKarlsruhesingletopteam,butwe’duseditbefore.

• Medianscanbeusedinsteadofmeans,andtheσ’sareRMS’softhe‐2lnQdistribu6on.

€

o − value =−2lnQ bkg − −2lnQ s+b( )

σ bkg2 +σ s+b

2

TomJunkBanffChallenge2 14

€

o − value =−2lnQ bkg − −2lnQ s+b( )

σ bkg2 +σ s+b

2

Problem <‐2lnQ>b RMSb <‐2lnQ>s+b RMSs+b o‐value

1nosyst 41.9 12.3 ‐46.3 14.3 4.7

2nosyst 19.1 8.6 ‐20.0 9.1 3.1

3nosyst 49.5 11.9 ‐68.9 19.5 5.2

123nosyst 110.6 19.1 ‐135.2 25.8 7.6

1syst 21.2 8.9 ‐28.1 13.5 3.0

2syst 12.8 6.6 ‐16.7 9.3 2.6

3syst 21.6 9.5 ‐25.6 12.2 3.0

123syst 55.5 14.6 ‐70.3 20.5 5.0

Toagoodapproxima6on,o‐valuesaddinquadratureforthecombina6on.Trueforthisproblem,butnottrueingeneral.

TomJunkBanffChallenge2 15

BackupMaterial

TomJunkBanffChallenge2 16

Fitting Nuisance Parameters to Reduce Sensitivity to Mismodeling

Means of PDF’s of -2lnQ very sensitive to background rate estimation.

Still some sensitivity in PDF’s residual due to prob. of each outcome varies with bg estimate.