CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on:...
Transcript of CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on:...
![Page 1: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/1.jpg)
CWoLa Hunting1
CWoLa Hunting:Extending the Bump Hunt with Machine Learning
Based on:Phys. Rev. Lett. 121, 241803 (2018)
[1805.02664] Jack Collins, Kiel Howe, Ben Nachman
![Page 2: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/2.jpg)
CWoLa Hunting2
Outline
1) Machine Learning
2) Model Unspecific Searches
3) CWoLa Hunting
![Page 3: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/3.jpg)
CWoLa Hunting3
Machine Learning
Classification
Regressionhttps://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
Classification
Generation
![Page 4: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/4.jpg)
CWoLa Hunting4
Machine Learning
Classification
Regression
https://research.nvidia.com/sites/default/files/publications/dnn_denoise_author.pdf
Regression
Generation
![Page 5: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/5.jpg)
CWoLa Hunting5
Machine Learning
Classification
Regression
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Generation
Generation
![Page 6: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/6.jpg)
CWoLa Hunting 6
![Page 7: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/7.jpg)
CWoLa Hunting 7
![Page 8: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/8.jpg)
CWoLa Hunting8
Machine Learning at the LHC
Classification
RegressionArXiv: 1511.05190 L. Oliveira, M. Kagan, L. Mackey, B. Nachman, A. Schwartzman
Classification: Jets
Generation
![Page 9: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/9.jpg)
CWoLa Hunting9
Machine Learning at the LHC
Classification
Regression
ArXiv:1707.08600. P. T. Komiske, E. M. Metodiev, B. Nachman, M. D. Schwartz
Regression: Pileup removal
Generation
![Page 10: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/10.jpg)
CWoLa Hunting10
Machine Learning at the LHC
Classification
Regression
ArXiv 1712.10321, M. Paganini, L. de Oliveira, B. Nachamn
Generation: Fast simulation
Generation
![Page 11: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/11.jpg)
CWoLa Hunting11
Basic Machine Learning Primer
https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
0) Decide objective: e.g. classify dog vs cat pictures1) Choose a network architecture: e.g. CNN2) Choose loss function (objective metric)3) Train network using training data4) Apply network on new test data
![Page 12: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/12.jpg)
CWoLa Hunting12
Basic Machine Learning Primer
https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
0) Decide objective: e.g. classify dog vs cat pictures1) Choose a network architecture: e.g. CNN2) Choose loss function (objective metric)3) Train network using training data4) Apply network on new test data
A NN is just a function mapping N inputnumbers to M output numbers.
Trainable internal numbers – weightsand biases – determine that function.
![Page 13: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/13.jpg)
CWoLa Hunting13
Basic Machine Learning Primer
https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
0) Decide objective: e.g. classify dog vs cat pictures1) Choose a network architecture: e.g. CNN2) Choose loss function (objective metric)3) Train network using training data4) Apply network on new test data
Naive example:Fraction of correct predictions
Practical example: Cross-entropy loss-∑ y(x) log[NN(x)]
![Page 14: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/14.jpg)
CWoLa Hunting14
Basic Machine Learning Primer
https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
0) Decide objective: e.g. classify dog vs cat pictures1) Choose a network architecture: e.g. CNN2) Choose loss function (objective metric)3) Train network using training data4) Apply network on new test data
Use some iterative optimizationalgorithm to minimize the lossfunction on training data.
Be careful in selecting training data!
![Page 15: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/15.jpg)
CWoLa Hunting15
Basic Machine Learning Primer
https://becominghuman.ai/building-an-image-classifier-using-deep-learning-in-python-totally-from-a-beginners-perspective-be8dbaf22dd8
0) Decide objective: e.g. classify dog vs cat pictures1) Choose a network architecture: e.g. CNN2) Choose loss function (objective metric)3) Train network using training data4) Apply network on new test data
Training: SlowTesting: Fast
Performance may be limited by thequality / relevance of training data.
![Page 16: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/16.jpg)
CWoLa Hunting16
BSM Searches: Nothing so far
![Page 17: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/17.jpg)
CWoLa Hunting17
BSM Searches: Nothing so far
Possibilities:
1) LHC doesn’t have the answers to our questions
2) Maybe new physics is rare: have to wait for high luminosity LHC
3) Maybe it is there but we are not doing the right search (theory bias has been wrong)
![Page 18: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/18.jpg)
CWoLa Hunting18
Are we missing something?
1) Ever-more sensitive dedicated searches for the standard culprits:
– Minimal Supersymmetry – Top Partners – diboson / ttbar resonances
![Page 19: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/19.jpg)
CWoLa Hunting19
Are we missing something?
1) Ever-more sensitive dedicated searches for the standard culprits:
– Minimal Supersymmetry – Top Partners – diboson / ttbar resonances
2) General-purpose ‘model-independent’ searches for unexpected new physics
![Page 20: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/20.jpg)
CWoLa Hunting20
Signatures vs Models
E.g. 2-body resonances (pp → X → SM SM):
[1610.09392] Craig, Draper, Kong, Ng, Whiteson
Signatures
Models
![Page 21: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/21.jpg)
CWoLa Hunting21
Signatures vs Models
E.g. 2-body resonances:
pp → X → BSM BSM largely uncovered
![Page 22: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/22.jpg)
CWoLa Hunting22
Basic Resonance Searches
E.g. Dijet Search
![Page 23: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/23.jpg)
CWoLa Hunting23
Basic Resonance Searches
Selection for signal-like events
E.g. Dijet Search E.g. WW resonance, tt resonance, etc.
![Page 24: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/24.jpg)
CWoLa Hunting24
Basic Resonance Searches
Selection for signal-like events
E.g. Dijet Search E.g. WW resonance, tt resonance, etc.
W-jets
‘Old fashioned’: Simple few-D substructure selection
‘Modern’: Deep NN classifier using ~few hundred jet constituent inputs (~300-D selection).
![Page 25: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/25.jpg)
CWoLa Hunting25
Basic Resonance Searches
Selection for signal-like events
E.g. Dijet Search E.g. ?? resonance
?-jets
![Page 26: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/26.jpg)
CWoLa Hunting26
A Traditional Dichotomy
Model Inclusive Search
– Weak signal assumptions– Basic selection criteria in few variables– Large backgrounds
– Risk missing a signal under background
Model Specific Search
– Strong signal assumptions– Sophisticated multivariate selection – Small backgrounds
– Risk not making the ‘correct’ signal selection
How to make a search with a sophisticated multivariate selection to beat backgrounds while using weak signal assumptions (unknown specific signal model)?
→ Learn selection from data
![Page 27: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/27.jpg)
CWoLa Hunting27
Why Train Machines on Data?
1) Maybe you have not simulated the correct signal model (either because you haven’t thought of it, or because it involves non-perturbative physics that prevents simulation)
![Page 28: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/28.jpg)
CWoLa Hunting28
Why Train Machines on Data?
Figure taken from Ben Nachman’s talk at BOOST 2018https://indico.cern.ch/event/649482/contributions/2993322/attachments/1688082/2715256/WeakSupervision_BOOST2018.pdf
1) Maybe you have not simulated the correct signal model (either because you haven’t thought of it, or because it involves non-perturbative physics that prevents simulation)
2) Monte-Carlo simulation for training data may differ from real LHC data
![Page 29: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/29.jpg)
CWoLa Hunting29
Weak Supervision
Solution for ML:
Train directly on data using mixed samples
A) LoLiProp (Learning from Labelled Proportions)Train using class proportions
[1702.00414] L. Dery, B. Nachman, F. Rubbo, A. Schwartzman[1706.09451] T. Cohen, M. Freytsis, B. Ostdiek
B) CWoLa (Classification Without Labels)
Train to classify as mixed sample 1 or 2.
[1708.02949] E. M. Metodiev, B. Nachman, J. Thaler
See
also
[180
1.10
158]
P. T
. Kom
iske
, E. M
. Met
odie
v, B
. Nac
hman
, M. D
. Sch
war
tz
![Page 30: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/30.jpg)
CWoLa Hunting30
CWoLa
CWoLa
Classifier trained to optimally discriminate mixed sample 1 from mixed sample 2 is also optimal for discriminating S from B, so long as:
– Samples 1 and 2 contain different fractions of S and B– S in sample 1 is drawn from the same distribution as S in sample 2– B in sample 1 is drawn from the same distribution as B in sample 2– Training statistics are sufficiently large
How to use this for a search where S is new physics and B is SM background?
![Page 31: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/31.jpg)
CWoLa Hunting31
CWoLa Hunting
1. Assume signal is localized in some specific variable in which background is smooth.
![Page 32: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/32.jpg)
CWoLa Hunting32
CWoLa Hunting
1. Assume signal is localized in some specific variable in which background is smooth.
2. Assume signal has some distinguishing characteristics within some broad set of additional observables y.
![Page 33: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/33.jpg)
CWoLa Hunting33
CWoLa Hunting
1. Assume signal is localized in some specific variable in which background is smooth.
2. Assume signal has some distinguishing characteristics within some broad set of additional observables y.
3. For some resonance mass hypothesis, split data into signal-region and sideband-region mixed samples
Mixed S
ample 1
Mixed S
ample 2
![Page 34: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/34.jpg)
CWoLa Hunting34
CWoLa Hunting
Train classifier to discriminate samples based on variables y
Note: background y distribution should not be strongly varying with the resonance variable.
Mixed S
ample 1
Mixed S
ample 2
Selection for signal-region-like
events
![Page 35: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/35.jpg)
CWoLa Hunting35
CWoLa Hunting
Mixed S
ample 1
Mixed S
ample 2
Selection for signal-region-like
events
1 2
![Page 36: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/36.jpg)
CWoLa Hunting36
Overfitting and the Look Elsewhere Effect
Of course, there is going to be a large trials factor, especially if y is high-dimensional.
Easy solution:Train test split(Statistical fluctuations in training and test set are uncorrelated)
More sophisticated:Nested cross-training
![Page 37: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/37.jpg)
CWoLa Hunting37
Nested Cross-Training
1) Divide entire dataset into k-folds
Test setTraining signal regionTraining sideband
![Page 38: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/38.jpg)
CWoLa Hunting38
Nested Cross-Training
2) Train CWoLa Classifiers
Train signal vs sideband k-1 times, rotating validation set.
Average the k-1 models to form an ensemble model
Background fluctuation contributions will destructively interfere, signal contributions constructively.
![Page 39: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/39.jpg)
CWoLa Hunting39
Nested Cross-Training
3) Select events in each k-fold and then merge
![Page 40: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/40.jpg)
CWoLa Hunting40
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
![Page 41: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/41.jpg)
CWoLa Hunting41
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
![Page 42: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/42.jpg)
CWoLa Hunting42
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
![Page 43: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/43.jpg)
CWoLa Hunting43
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
2σ
![Page 44: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/44.jpg)
CWoLa Hunting44
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
2σ
3.5σ
![Page 45: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/45.jpg)
CWoLa Hunting45
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
2σ
3.5σ
7σ
![Page 46: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/46.jpg)
CWoLa Hunting46
Application to Bump Hunt
In signal region:S = 522,S/B = 0.64%
1.5σ
2σ
3.5σ
7σ
![Page 47: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/47.jpg)
CWoLa Hunting47
What has the machine learnt?
Jet 1 Jet 2
Low-ish particle multiplicity
High mass
4-prongy
2-prongy
Moderate mass
Low-ish particle multiplicity
![Page 48: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/48.jpg)
CWoLa Hunting48
No Signal → No Bump!
![Page 49: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/49.jpg)
CWoLa Hunting49
What has the machine learnt?
Jet 1 Jet 2
Nothing, as desired!
![Page 50: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/50.jpg)
CWoLa Hunting50
Mass Scan
![Page 51: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/51.jpg)
CWoLa Hunting51
Mass Scan
![Page 52: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/52.jpg)
CWoLa Hunting52
Mass Scan
![Page 53: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/53.jpg)
CWoLa Hunting53
Mass Scan
![Page 54: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/54.jpg)
CWoLa Hunting54
Mass Scan
![Page 55: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/55.jpg)
CWoLa Hunting55
Mass Scan
![Page 56: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/56.jpg)
CWoLa Hunting56
Mass Scan
![Page 57: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/57.jpg)
CWoLa Hunting57
Mass Scan
![Page 58: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/58.jpg)
CWoLa Hunting58
Mass Scan
![Page 59: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/59.jpg)
CWoLa Hunting59
Mass Scan
![Page 60: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/60.jpg)
CWoLa Hunting60
Performance Comparison
Better
Fully supervised ‘dedicated search’
Fully supervised, wrong model.
![Page 61: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/61.jpg)
CWoLa Hunting61
General CWoLa Hunting
● Need some variable X (e.g. m_JJ) in which bg is smooth and signal is localized
● Need some other variables {Y} (e.g. jet substructure) which may provide discriminating power which may be a-priori unknown.
● {Y} should not be strongly correlated with X over the X-width of the signal.
● Or alternatively, if correlated, there may be a way to decorrelate (e.g. if we can predict or measure the correlation, that can be subtracted away to create new uncorrelated variables).
● Can we use low level inputs rather than expert variables?– Difficult to decorrelate auxiliary variables from resonance variable, but there are
ways.– Pessimist: Only O(100) signal events → not enough to train with.– But can’t know until someone tries it!
![Page 62: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/62.jpg)
CWoLa Hunting62
Other work: Autoencoders
Train only on ‘background’ (no need for signal training)
Can reconstruct typical QCD background jets well, but atypical jets poorly.
→ Classify as ‘signal-like’ jets with poor reconstruction loss.
Advantage: no need for signal events for training.
Disadvantage: Can’t make use of specific signal characteristics for selection
[1808.08992] M. Farina, Y. Nakai, D. Shih
![Page 63: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/63.jpg)
CWoLa Hunting63
Background-only training vs signal/sideband:
Background-only Signal / Sideband
Tagger performance does not depend on signal statistics.
Tagger can never learn the specific peculiar features of the signal, and so cannot improve with greater signal rate.
Tagger relies on there being sufficient signal statistics for training.
Tagger can learn the specific peculiar features of the signal, and so improves with greater signal rate, and allows for signal characterization.
Stronger in limit of very low signal statistics
Stronger in limit of very high signal statistics??
![Page 64: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/64.jpg)
CWoLa Hunting 64
![Page 65: CWoLa HuntingCWoLa Hunting 1 CWoLa Hunting: Extending the Bump Hunt with Machine Learning Based on: Phys. Rev. Lett. 121, 241803 (2018) [1805.02664] Jack Collins, Kiel Howe, Ben Nachman](https://reader033.fdocuments.us/reader033/viewer/2022051207/6039809294c5a174314c3b62/html5/thumbnails/65.jpg)
CWoLa Hunting65
Toy Statistics