MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and...

48
MACHINE LEARNING IN HIGH ENERGY PHYSICS LECTURE #1 Alex Rogozhnikov, 2015

Transcript of MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and...

Page 1: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

MACHINE LEARNING IN HIGHENERGY PHYSICSLECTURE #1

Alex Rogozhnikov, 2015

Page 2: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

INTRO NOTES4 daystwo lectures, two practice seminars every daythis is introductory track to machine learningkaggle competition!

Page 3: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

WHAT IS ML ABOUT?Inference of statistical dependencies which give us ability to

predict

Data is cheap, knowledge is precious

Page 4: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

WHERE ML IS CURRENTLY USED?Search engines, spam detectionSecurity: virus detection, DDOS defenseComputer vision and speech recognitionMarket basket analysis, Customer relationship management(CRM)Credit scoring, fraud detectionHealth monitoringChurn prediction... and hundreds more

Page 5: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

ML IN HIGH ENERGY PHYSICSHigh-level triggers (LHCb trigger system: )Particle identificationTaggingStripping lineAnalysis

40MHz → 5kHz

Different data is used on different stages

Page 6: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

GENERAL NOTIONIn supervised learning the training data is represented as setof pairs

,xi yi

is index of event is vector of features available for event is target — the value we need to predict

ixiyi

Page 7: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

CLASSIFICATION EXAMPLE

, if finite set

on the plot: ,

Examples:

defining type of particle (or decaychannel)

— binary classification, 1is signal, 0 is bck

! Yyi Y

!xi ℝ2 ! {0, 1, 2}yi

Y = {0, 1}

Page 8: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

REGRESSIONy ! ℝ

Examples:

predicting price of house by it's positionspredicting number of customers / money incomereconstructing real momentum of particle

Why need automatic classification/regression?

in applications up to thousands of featureshigher qualitymuch faster adaptation to new problems

Page 9: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

CLASSIFICATION BASED ON NEAREST NEIGHBOURSGiven training set of objects and their labels wepredict the label for new observation.

{ , }xi yi

= , j = arg ρ(x, )y ̂ yj mini

xi

Page 10: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

VISUALIZATION OF DECISION RULE

Page 11: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

NEAREST NEIGHBOURSkA better way is to use neighbours:k

(x) =pi# of knn events in class i

k

Page 12: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

k = 1, 2, 5, 30

Page 13: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

OVERFITTINGwhat is the quality of classification on training dataset when

?answer: it is ideal (closest neighbor is event itself)quality is lower when this doesn't mean is the best, it means we cannot use training events to estimate qualitywhen classifier's decision rule is too complex and capturesdetails from training data that are not relevant todistribution, we call this overfitting (more details tomorrow)

k = 1

k > 1k = 1

Page 14: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

KNN REGRESSORRegression with nearest neighbours is done by averaging ofoutput

=y ̂ 1k ∑

j!knn(x)

yj

Page 15: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

KNN WITH WEIGHTS

Page 16: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations
Page 17: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

COMPUTATIONAL COMPLEXITYGiven that dimensionality of space is and there are training samples:

d n

training time ~ O(save a link to data)prediction time: for each samplen × d

Page 18: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

SPACIAL INDEX: BALL TREE

Page 19: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

BALL TREEtraining time ~ prediction time ~ for each sample

O(d × n log(n))log(n) × d

Other option exist: KD-tree.

Page 20: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

OVERVIEW OF KNN1. Awesomely simple classifier and regressor2. Have too optimistic quality on training data3. Quite slow, though optimizations exist4. Hard times with data of high dimensions5. Too sensitive to scale of features

Page 21: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

SENSITIVITY TO SCALE OF FEATURESEuclidean distance:ρ(x, y = ( + + ( + + ⋯ + ( +)2 x1 y1)2 x2 y2)2 xd yd )2

Change scale fo first feature:ρ(x, y = (10 + 10 + ( + + ⋯ + ( +)2 x1 y1)2 x2 y2)2 xd yd )2

ρ(x, y U 100( +)2 x1 y1)2

Scaling of features frequently increases quality.

Page 22: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

DISTANCE FUNCTION MATTERSMinkowski distance

Canberra

Cosine metric

(x, y) = ( +ρp *i xi yi)p

ρ(x, y) = *i| + |xi yi

| | + | |xi yi

ρ(x, y) =< x, y >

|x| |y|

Page 23: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

MINUTES BREAKx

Page 24: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

RECAPITULATION1. Statistical ML: problems2. ML in HEP3. nearest neighbours classifier and regressor.k

Page 25: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

MEASURING QUALITY OF BINARYCLASSIFICATIONThe classifier's output in binary classification is real variable

Which classifier is better?All of them are identical

Page 26: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

ROC CURVE

These distributions have the same ROC curve: (ROC curve is passed signal vs passed bck dependency)

Page 27: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

ROC CURVE DEMONSTRATION

Page 28: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

ROC CURVEContains important information: all possible combinations of signal and backgroundefficiencies you may achieve by setting thresholdParticular values of thresholds (and initial pdfs) don'tmatter, ROC curve doesn't contain this informationROC curve = information about order of events:

Comparison of algorithms should be based on informationfrom ROC curve

s s b s b ... b b s b b

Page 29: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

TERMINOLOGY AND CONVENTIONSfpr = background efficiency = btpr = signal efficiency = s

→

Page 30: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

where are predictions ofrandom background and signal events.

ROC AUC (AREA UNDER THE ROC CURVE)

ROC AUC = P(x < y) x, y

Page 31: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

Which classifier is better for triggers? (they have the same ROC AUC)

Page 32: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

STATISTICAL MACHINE LEARNINGMachine learning we use in practice is based on statistics

1. Main assumption: the data is generated from probabilisticdistribution:

2. Does there really exist the distribution of people / pages?3. In HEP these distributions do exist

p(x, y)

Page 33: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

OPTIMAL CLASSIFICATION. OPTIMALBAYESIAN CLASSIFIERAssuming that we know real distributions wereconstruct using Bayes' rule

p(x, y)

p(y|x) = =p(x, y)p(x)

p(y)p(x|y)p(x)

=p(y = 1 | x)p(y = 0 | x)

p(y = 1) p(x | y = 1)p(y = 0) p(x | y = 0)

LEMMA (NEYMAN–PEARSON):The best classification quality is provided by

p(y = 1 | x)

Page 34: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

The best classification quality is provided by

(optimal bayesian classifier)p(y = 0 | x)

OPTIMAL BINARY CLASSIFICATIONOptimal bayesian classifier has highest possible ROC curve.Since the classification quality depends only on order,

gives optimal classification quality too!p(y = 1 | x)

=p(y = 1 | x)p(y = 0 | x)

p(y = 1) p(x | y = 1)p(y = 0) p(x | y = 0)

Page 35: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

FISHER'S QDA (QUADRATIC DISCRIMINANTANALYSIS)Reconstructing probabilities fromdata, assuming those are multidimensional normaldistributions:

p(x | y = 1), p(x | y = 0)

p(x | y = 0) U ( , )μ0 Σ0

p(x | y = 1) U ( , )μ1 Σ1

Page 36: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations
Page 37: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

QDA COMPLEXITY samples, dimensionsn d

training takes

computing covariation matrix

inverting covariation matrix

prediction takes for each sample

O(n + )d2 d3

O(n )d2

O( )d3

O( )d2

f (x) = exp (+ (x + μ (x + μ))1(2π |Σ)k/2 |1/2

12

)TΣ+1

Page 38: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

QDAsimple decision rulefast predictionmany parameters to reconstruct in high dimensionsdata almost never has gaussian distribution

Page 39: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

WHAT ARE THE PROBLEMS WITHGENERATIVE APPROACH?Generative approach: trying to reconstruct , then useit to predict.

p(x, y)

Real life distributions hardly can be reconstructedEspecially in high-dimensional spacesSo, we switch to discriminative approach: guessing p(y|x)

Page 40: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

This is (finding parameters ).

LINEAR DECISION RULE

Decision function is linear:

d(x) =< w, x > +w0

{ d(x) > 0,d(x) < 0,

 class  + 1 class  + 1

parametric model w, w0

Page 41: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

FINDING OPTIMAL PARAMETERSA good initial guess: get such , that error ofclassification is minimal ([true] = 1, [false] = 0):

Discontinuous optimization (arrrrgh!)Let's make decision rule smooth

w, w0

= [ y sgn(d( ))]∑i!events

yi xi

(x)p+1

(x)p+1

= f (d(x))= 1 + (x)p+1

⎧

⎩⎨⎪⎪

f (0) = 0.5f (x) > 0.5f (x) < 0.5

if x > 0if x < 0

Page 42: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

LOGISTIC FUNCTIONa smooth step rule.

σ(x) = =ex

1 + ex1

1 + e+x

PROPERTIES1. monotonic, 2. 3. 4.

σ(x) ! (0, 1)σ(x) + σ(+x) = 1

(x) = σ(x)(1 + σ(x))σ ′

2 σ(x) = 1 + tanh(x/2)

Page 43: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

LOGISTIC FUNCTION

Page 44: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

Optimizing log-likelihood (with probabilities obtained withlogistic function)

LOGISTIC REGRESSION

d(x)(x)p+1

(x)p+1

===

< w, x > +w0

σ(d(x))σ(+d(x))

= + ln( ( )) = L( , ) → min1N ∑

i!events

pyixi

1N ∑

i

xi yi

Page 45: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

Exercise: find expression and build plot for L( , )xi yi

DATA SCIENTIST PIPELINE

1. Experiments in appropriate high-level language orenvironment

2. After experiments are over — implement final algorithm inlow-level language (C++, CUDA, FPGA)

Second point is not always needed.

Page 46: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

SCIENTIFIC PYTHONNumPy vectorized computations in python

Matplotlib for drawing

Pandas for data manipulation and analysis (based onNumPy)

Page 47: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

SCIENTIFIC PYTHON

Scikit-learn most popular library for machine learning

Scipy libraries for science and engineering

Root_numpy convenient way to work with ROOT files

Page 48: MACHINE LEARNING IN HIGH ENERGY PHYSICS...OVERVIEW OF KNN 1. Awesomely simple classifier and regressor 2. Have too optimistic quality on training data 3. Quite slow, though optimizations

THE END


AENC Awesomely Simple

AENC Awesomely Simple

Earthquake prediction model using support vector regressor ......RESEARCH ARTICLE Earthquake prediction model using support vector regressor and hybrid neural networks Khawaja M. Asim1*,

Earthquake prediction model using support vector regressor ......RESEARCH ARTICLE Earthquake prediction model using support vector regressor and hybrid neural networks Khawaja M. Asim1*,

ISPA Awesomely Simple keynote 3.15

ISPA Awesomely Simple keynote 3.15

kNN, LVQ, SOM

kNN, LVQ, SOM

The Regressor | September, Issue 1

The Regressor | September, Issue 1

DX140W - KNN Cambodia

DX140W - KNN Cambodia

O2E Awesomely Simple Keynote 2.19.15

O2E Awesomely Simple Keynote 2.19.15

Regression with One Regressor: Hypothesis Tests and Confidence Intervals

Regression with One Regressor: Hypothesis Tests and Confidence Intervals

Sweet KNN: An Efficient KNN on GPU through Reconciliation ... · show that Sweet KNN outperforms existing GPU implementations on KNN by up to 120X (11X on average). I. INTRODUCTION

Sweet KNN: An Efficient KNN on GPU through Reconciliation ... · show that Sweet KNN outperforms existing GPU implementations on KNN by up to 120X (11X on average). I. INTRODUCTION

Lecture 11: Cross validation - Stanford UniversityScenario2 KNN!1 KNN!CV LDA Logistic QDA 0.25 0.30 0.35 0.40 0.45 SCENARIO 1 KNN!1 KNN!CV LDA Logistic QDA 0.15 0.20 0.25 0.30 SCENARIO

Lecture 11: Cross validation - Stanford UniversityScenario2 KNN!1 KNN!CV LDA Logistic QDA 0.25 0.30 0.35 0.40 0.45 SCENARIO 1 KNN!1 KNN!CV LDA Logistic QDA 0.15 0.20 0.25 0.30 SCENARIO

CHAPTER 4 Linear Regression with One Regressor

CHAPTER 4 Linear Regression with One Regressor

KNN NSU H APER

KNN NSU H APER

Knn solution

Knn solution

12 Knn Perceptron

12 Knn Perceptron

More Creatures & Characters: Drawing Awesomely Wild ......More Creatures & Characters: Drawing Awesomely Wild, Wacky and Funny Animals - Schiffer Publishing 60 D Title MCandC-Animals

More Creatures & Characters: Drawing Awesomely Wild ......More Creatures & Characters: Drawing Awesomely Wild, Wacky and Funny Animals - Schiffer Publishing 60 D Title MCandC-Animals

KNN News 042715

KNN News 042715

Linear Regression with One Regressor - mysmu.edu · Linear Regression with One Regressor (Stock/Watson Chapter 4) Outline ... 2.28 STR • Districts with ... Root MSE = 18.581 ...

Linear Regression with One Regressor - mysmu.edu · Linear Regression with One Regressor (Stock/Watson Chapter 4) Outline ... 2.28 STR • Districts with ... Root MSE = 18.581 ...

kNN Imputation

kNN Imputation

kNN & Naïve Bayes

kNN & Naïve Bayes

Rooftop Solar-KNN

Rooftop Solar-KNN