Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See...

38
Conclusions from TrackML the HEP Tracking Machine Learning Challenge Organization Jean-Roch Vlimant (Caltech) Vincenzo Innocente, Andreas Salzburger (CERN) David Rousseau, Yetkin Yilmaz (LAL-Orsay) Paolo Calafiura, Steven Farrell, Heather Gray (LBNL) Vladimir Gligorov (LPNHE-Paris) Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University) Laurent Basara , Cécile Germain, Isabelle Guyon, Victor Estrade (LRI-Orsay) Edward Moyse (University of Massachussets) Mikhail Hushchyn, Andrey Ustyuzhanin (Yandex, HSE)

Transcript of Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See...

Page 1: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

Conclusions from TrackMLthe HEP Tracking

Machine Learning Challenge

Organization

Jean-Roch Vlimant (Caltech)

Vincenzo Innocente, Andreas Salzburger (CERN)

David Rousseau, Yetkin Yilmaz (LAL-Orsay)

Paolo Calafiura, Steven Farrell, Heather Gray (LBNL)

Vladimir Gligorov (LPNHE-Paris)

Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University)

Laurent Basara, Cécile Germain, Isabelle Guyon, Victor Estrade (LRI-Orsay)

Edward Moyse (University of Massachussets)

Mikhail Hushchyn, Andrey Ustyuzhanin (Yandex, HSE)

Page 2: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 2

Tracking crisis

● Tracking dominates reconstruction CPU time

● At best quadratic● HL-LHC (2025) :

unmanageable ● Everything tried?

→ TrackML challenge

Page 3: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 3

Page 4: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 4

See also in outreach session talk by D. Rousseau« TrackML : the roller coaster of organizing a HEP challenge on Kaggle and Codalab » https://indico.cern.ch/event/577856/contributions/3423422/

Page 5: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 5

See also in outreach session talk by D. Rousseau« TrackML : the roller coaster of organizing a HEP challenge on Kaggle and Codalab » https://indico.cern.ch/event/577856/contributions/3423422/

Page 6: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 6

First phase :Accuracy May – August 2018

Page 7: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 7

Leaderboard scores

Page 8: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 8

Optimizing score optimizes physics

Page 9: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 9

Participants dendrogram

Page 10: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 10

Phase 1 winner : Top Quarks

Page 11: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 11

Phase 1 #2 : outrunner

● Train DNN on hit pairs

▸ 27 inputs (x,y,z,cells,…)

▸ 4k-2k-2k-2k-1k hidden layers

● Compute full hit adjacency matrix:

▸ probability P(i,j) that 2 hits match

▸ Pick high probability comb

● True Deep Learning Solution

▸ No track following

▸ No geometric modelling

● 1 Day / event

Page 12: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 12

Second phase :ThroughputOct 2018 – March 2019

Page 13: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 13

Leaderboard evolution

Page 14: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 14

Leaderboard evolution

Page 15: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 15

Leaderboard evolution

Page 16: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 16

Final leaderboard

Page 17: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 17

#1 S. Gorbunov: « fast combinatorial »

https://indico.cern.ch/event/813759/contributions/3479705/attachments/1871795/3080436/trackML_Gorbunov-CERN2019.pdf

Page 18: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 18

#2 FASTrack: Graph of neighbours, cellular automata and Kalman filter

https://indico.cern.ch/event/813759/contributions/3479706/attachments/1870758/3078234/TheTrackML_workshop_talk.pdf

Page 19: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 19https://indico.cern.ch/event/813759/contributions/3479709/attachments/1871944/3080707/TrackML.Kunze.pdf

Page 20: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 20

Conclusions

● Open tracking competition organised to reach out to CS and ML communities

● Winner and runner-up HEP tracking experts…

● Retained solution will be blend from HEP expertise and new ideas

● Dataset released on CERN Open Data Portalto serve as benchmark

● Ongoing work

Page 21: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 21

Contacts

● Contact : [email protected]● https://sites.google.com/site/trackmlparticle● Twitter : @trackmllhc● Accuracy phase @ Kaggle:

https://www.kaggle.com/c/trackml-particle-identification▸ Chapter in the NeurIPS2018 Competition book arXiv:1904.06778

● Throughput phase @ Codalab: https://competitions.codalab.org/competitions/20112

▸ Write-up to be finalised

● TrackML challenge Grand Finale: https://indico.cern.ch/event/813759/

Page 22: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

Conclusions from TrackMLthe HEP Tracking

Machine Learning Challenge

Organization

Jean-Roch Vlimant (Caltech)

Vincenzo Innocente, Andreas Salzburger (CERN)

David Rousseau, Yetkin Yilmaz (LAL-Orsay)

Paolo Calafiura, Steven Farrell, Heather Gray (LBNL)

Vladimir Gligorov (LPNHE-Paris)

Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University)

Laurent Basara, Cécile Germain, Isabelle Guyon, Victor Estrade (LRI-Orsay)

Edward Moyse (University of Massachussets)

Mikhail Hushchyn, Andrey Ustyuzhanin (Yandex, HSE)

Page 23: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 23

Page 24: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 24

Page 25: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 25

Winning solution not ML→

Backup

Page 26: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 26

Winning solution not ML→

Page 27: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 27

Winning solution not ML→

Page 28: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 28

Winning solution not ML→

Page 29: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 29

Winning solution not ML→

Page 30: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 30

Yuval & Trian (#7)

Page 31: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 31

Page 32: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 32

Page 33: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 33

Page 34: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 34

Other contestants● #7 : Yuval & Trian

▸ « Binned randomized Hough transform » for clustering

▸ ML (LightGBM) to merge tracks

● # 9 : CPMP - « DBSCAN Forever »▸ DBSCAN on transformed space including deviation from

helix

▸ On each iteration clusters = new candidate tracks, merged

● #12 : The Finnies▸ DBSCAN variants 5 hit track seeding→

▸ LSTM estimate 10 hits→

▸ KNN for Track fitting

Page 35: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 35

Majority particle(truth)

Examined track(reconstructed)

Nmaj N

reco

Ngood

Puritymaj

= Ngood

/ Nmaj

Purityreco

= Ngood

/ Nreco

Scoring

Page 36: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 36

Page 37: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 37

« passing through origin »

Scattering in detector

« homogeneous magnetic field »

Energy (hence momentum) loss

Page 38: Conclusions from TrackML the HEP Tracking Machine Learning ...€¦ · TrackML conclusion 4 See also in outreach session talk by D. Rousseau « TrackML : the roller coaster of organizing

TrackML conclusion 38