Charles Plager UCLA LJ+MET Meeting March 23, 2008

Charles PlagerCharles Plager LJ+MET, March 23, 2009LJ+MET, March 23, 2009 Page 1

Charles PlagerCharles PlagerUCLAUCLA

LJ+MET MeetingLJ+MET MeetingMarch 23, 2008March 23, 2008

““Throwing PEs” Throwing PEs” and Moreand More


““PE? What’s a PE?”PE? What’s a PE?”

• Pseudo-experiment (PE) is the process of generating a set of random Pseudo-experiment (PE) is the process of generating a set of random numbers that simulates the numbers that simulates the given quantitiesgiven quantities needed to perform an needed to perform an analysis.analysis.– This can be as simple of generating a number of events for a This can be as simple of generating a number of events for a

counting experiment.counting experiment.– This can be as complicated as picking a subset of This can be as complicated as picking a subset of “events”“events”

needed to perform correlated analyses.needed to perform correlated analyses.• In In Charles-speakCharles-speak::

– PEPE )) One pseudo-experiment. One pseudo-experiment.– PEsPEs )) Many pseudo-experiments. Many pseudo-experiments.– Throw PEsThrow PEs )) Generate many pseudo-experiments Generate many pseudo-experiments

• ““Why throw PEs?”Why throw PEs?”– Throwing and Throwing and “fitting”“fitting” PEs is a very useful way of: PEs is a very useful way of:

• Debugging an analysisDebugging an analysis• Checking correlations between analysesChecking correlations between analyses• Measuring sensitivity of a given analysisMeasuring sensitivity of a given analysis


Simple Counting Experiment PESimple Counting Experiment PE

– You expect You expect NNbackback background events and background events and NNsigsig signal events. signal events.

– For each PE, your number of observed events:For each PE, your number of observed events:

nObsnObs = random.Poisson (N= random.Poisson (Nbackback) + random.Poisson (N) + random.Poisson (Nsigsig))

= random.Poisson (N= random.Poisson (Nback + back + NNsigsig))

• Your analysis is a simple Your analysis is a simple counting experiment with no counting experiment with no systematic uncertainties.systematic uncertainties.


PEs from TemplatesPEs from Templates

• Same idea as “Simple Counting PEs,” except that we can’t Same idea as “Simple Counting PEs,” except that we can’t merge merge signal and background:signal and background:

– NNobsSigobsSig = random.Poisson (N = random.Poisson (Nsigsig))

– NNobsBackobsBack = random.Poisson (N = random.Poisson (Nbackback))

• For each signal event, a value is drawn from the templateFor each signal event, a value is drawn from the template• Fit the resulting PE as if it were data.Fit the resulting PE as if it were data.


Aside: PEs with a Fixed Number of EventsAside: PEs with a Fixed Number of Events

• It may sometimes happen that one wants to generate PEs with a fixed It may sometimes happen that one wants to generate PEs with a fixed number of events, number of events, NNtotaltotal..

– E.g.,E.g., one wants to match the total number of events in a particular one wants to match the total number of events in a particular datadata sample.sample.

• If you are assuming, for example, a small background If you are assuming, for example, a small background NNbackback and a large and a large

signal signal NNsigsig, you , you can notcan not::

– Poisson fluctuate background Poisson fluctuate background NNobsBackobsBack = Poisson (N = Poisson (Nbackback), and then), and then

– NNobsSigobsSig = N = Ntotaltotal – N – Nbackback

• You need to fluctuate both:You need to fluctuate both:

– NNobsBackobsBack = Poisson (N = Poisson (Nbackback),),

– NNobsSigobsSig = Poisson (N = Poisson (Nsigsig),),

– Keep only if Keep only if NNobsBackobsBack + + NNobsSig obsSig == == NNtotaltotal

• Note: If you have Note: If you have NNobsBackobsBack + + NNobsSig obsSig > > NNtotaltotal, ,

– Pick random event to get rid of (signal Pick random event to get rid of (signal or background)or background)

– Repeat until Repeat until NNobsBackobsBack + + NNobsSig obsSig == == NNtotaltotal


PEs with Systematic UncertaintiesPEs with Systematic Uncertainties

• ““Blur, then throw”Blur, then throw”• You have a whole bunch of estimates described by Gaussian You have a whole bunch of estimates described by Gaussian

uncertainties: x uncertainties: x §§ xx..

• For each PE, generate a value for each estimate from the correct For each PE, generate a value for each estimate from the correct Gaussian distribution.Gaussian distribution.

• Make sure all estimates are physical. If not, repeat previous stepMake sure all estimates are physical. If not, repeat previous step**..• Given this set of estimates, generate PE as we have done in previous Given this set of estimates, generate PE as we have done in previous

steps.steps.


Correlated PEsCorrelated PEs

• ““We have two analyses that use overlapping data samples. What do we We have two analyses that use overlapping data samples. What do we do?”do?”

• Instead of picking values from a template, we pick whole “Instead of picking values from a template, we pick whole “events”events” from MC.from MC.– All analyses then take what they need from each event.All analyses then take what they need from each event.

• Using weighted MC is more complicated here:Using weighted MC is more complicated here:– Pick eventsPick events– Generate random number to see if event is keptGenerate random number to see if event is kept– Repeat until total number of events is reached in sub-sampleRepeat until total number of events is reached in sub-sample– ImportantImportant: Reweighting is very powerful technique and throwing PEs is : Reweighting is very powerful technique and throwing PEs is

a solved problem (see me).a solved problem (see me).


Your Analysis CheckupYour Analysis Checkup

• I’ve now thrown many PEs. What do I look at?I’ve now thrown many PEs. What do I look at?

• For each PE, take the results and treat it like you would with data:For each PE, take the results and treat it like you would with data:– Fit, etc.Fit, etc.

• Look at different distributions:Look at different distributions:– Uncertainties Uncertainties )) Sensitivity Sensitivity – Pull distributions (more later)Pull distributions (more later)

• Looking for coverage, bias, Looking for coverage, bias, etc.etc.

• Hint: Store PE results in a Hint: Store PE results in a TTreeTTree• Easier to debug problems.Easier to debug problems.


Pull DistributionsPull Distributions

• A pull distribution is a very useful way to make sure your analysis A pull distribution is a very useful way to make sure your analysis machinery is doing what it is supposed to.machinery is doing what it is supposed to.– Checks for Checks for biasesbiases as well as as well as underunder and and overover coverage. coverage.

– For each event, calculate:For each event, calculate:– Resulting distribution should be a unit Gaussian.Resulting distribution should be a unit Gaussian.


More with Pull DistributionsMore with Pull Distributions

• If you measure asymmetric uncertainties, simply use the If you measure asymmetric uncertainties, simply use the “right one:”“right one:”

– upperupper : measurement < true value : measurement < true value

– lower lower : measurement > true value : measurement > true value

• If the parameter of interest has a Gaussian constraint in your fit:If the parameter of interest has a Gaussian constraint in your fit:

– If you have Gaussian constraints in your fit, it is important to throw If you have Gaussian constraints in your fit, it is important to throw the PEs using the same distribution. the PEs using the same distribution. )) Throw Consistently Throw Consistently..

• Remember: checking pull distribution is a Remember: checking pull distribution is a necessarynecessary, but not , but not sufficient checksufficient check..

MeasLower Upper

TrueTrue


HintsHints

• Small statistics can sometimes Small statistics can sometimes “mess” “mess” with pull distributions:with pull distributions:– If you aren’t already, try varying the If you aren’t already, try varying the

true value of the variable when true value of the variable when throwing the PE.throwing the PE.

– E.g.,E.g., When throwing pull distributions When throwing pull distributions for top pair cross sections, vary the for top pair cross sections, vary the theoretical cross section using the theoretical cross section using the theory errors.theory errors.

• Root lets you generate random numbers Root lets you generate random numbers from a histogram:from a histogram:

)) value = myHistPtr->Random()value = myHistPtr->Random()


Root and Random Number GeneratorsRoot and Random Number Generators

• For all modern versions of Root (≥ 5.18), For all modern versions of Root (≥ 5.18), gRandomgRandom is an instance of is an instance of TRandom3TRandom3. . – This is a decent random number generator.This is a decent random number generator.

• If you are using an older version of Root, this may not be the default.If you are using an older version of Root, this may not be the default.– Change it:Change it:delete gRandom; gRandom = new TRandom3;delete gRandom; gRandom = new TRandom3;

• If you want to run PE jobs in parallel, If you want to run PE jobs in parallel, make sure you set the jobs to have make sure you set the jobs to have different random seeds:different random seeds:– gRandom->SetSeed (seed);gRandom->SetSeed (seed);

wherewhereseed is an integer not equal to 0.seed is an integer not equal to 0.

• If you don’t set different seeds, yourIf you don’t set different seeds, yourdifferent jobs will not generate differentdifferent jobs will not generate differentPEs.PEs.

Hardware Random Number Generator


Warning:Warning:

• The concept of ensembles (The concept of ensembles (e.g.,e.g., PEs) is the heart of PEs) is the heart of frequentist frequentist statisticalstatistical thought. thought. – When I talk about how to deal with systematic uncertainty, I will be When I talk about how to deal with systematic uncertainty, I will be

effectively talking about effectively talking about integratingintegrating over systematic priors. over systematic priors.

)) Bayesian statisticsBayesian statistics..

• Most of this will be a (reasonable) mix of the Most of this will be a (reasonable) mix of the two.two.– These methods will be sufficient for almost These methods will be sufficient for almost

all of your needs, all of your needs, butbut::– These methods, however, will not always These methods, however, will not always

lead to proper frequentist coverage.lead to proper frequentist coverage.

?


SummarySummary

• Throwing PEs is fun, easy, and important.Throwing PEs is fun, easy, and important.– Weights and reweighting is cool and powerful.Weights and reweighting is cool and powerful.

• Pull distributions should be standard operating procedure.Pull distributions should be standard operating procedure.• Using these simple ideas can really pay off in the end!Using these simple ideas can really pay off in the end!

Charles Plager UCLA LJ+MET Meeting March 23, 2008

Documents

Transcript of Charles Plager UCLA LJ+MET Meeting March 23, 2008