STAT2450 - Introduction to Data Mining with R Vector...
Transcript of STAT2450 - Introduction to Data Mining with R Vector...
![Page 1: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/1.jpg)
Lecture #10: Introduction to Support
Vector MachinesMat Kallada
STAT2450 - Introduction to Data Mining with R
![Page 2: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/2.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines
Multi-class Support Vector Machines
Kernels and Support Vector Machines
Support Vector Machines for Regression
![Page 3: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/3.jpg)
Data Mining Classifiers to Play Go: Google’s AlphaGo
![Page 4: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/4.jpg)
Remember - Deep Blue’s Win in 1997?
They didn’t use data mining
But still cool
![Page 5: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/5.jpg)
Data Visualization Strategies
We’ve seen five so far:
- Scatter Plots: Data Points on Cartesian Plane
- Line Plots: Change of Numerical value against Numerical value
- Bar Graphs: Categorical against Numerical values
- Histograms: Count distribution of values
- Heatmaps: Categorical variable against another Categorical
Variables
![Page 6: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/6.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines ←
Multi-class Support Vector Machines
Kernels and Support Vector Machines
Support Vector Machines for Regression
![Page 7: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/7.jpg)
Let’s change gears for a bit...
Remember, We learned two ways to draw lines
To solve regression or classification tasks
![Page 8: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/8.jpg)
Ways to Create Predictive Models(I.e. Methods to solve the Supervised Data Mining Setup)
Decision Trees
Construct a decision tree which chops on the vector space
“Chops” are feature splits which minimize error
K-Nearest Neighbours
Look at the K-closest Points in Training Data
![Page 9: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/9.jpg)
Supervised Data Mining: The Line Drawing ContestWho can draw the most “realistic” line?
![Page 10: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/10.jpg)
Supervised Data Mining: The Line Drawing ContestWho can draw the most “realistic” line?
K-nearest Neighbours
Decision Trees (“rpart”)
Grid
K = 5 to 10
Grid
cp = 0.01 to 0.10
Hei
ght
Width
Hei
ght
Width
Data (in a table form)
![Page 11: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/11.jpg)
Supervised Data Mining: The Line Drawing ContestTo evaluate whether these lines/curves actually work
Let’s use the one with highest performance
We need to use either hold-out validation or K-Fold Cross-Validation
Test Set It got 99.5% right on test set!
Model Built with K=3
![Page 12: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/12.jpg)
Supervised Data Mining: The Line Drawing ContestTo evaluate whether these lines/curves actually work
Let’s use the one with highest performance
We need to use either hold-out validation or K-Fold Cross-Validation
It has a 3-Fold Cross
Validation Estimate of
98.6%
Testing Set
Training Set
Testing Set
Training Set
Training Set
Training Set
Training Set
Testing Set
Training Set
Hei
ght
Width
Hei
ght
Width
Hei
ght
Width
Use all data to build a predictive model with KNN and K=5
3 Models built with KNN=5
Expect it to work 98.6% of time
Hei
ght
WidthFinal Model you use in real-world
![Page 13: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/13.jpg)
Supervised Data Mining: The Line Drawing ContestTo evaluate whether these lines/curves actually work
Let’s use the one with highest performance
We need to use either hold-out validation or K-Fold Cross-Validation
Both work but K-fold Cross-Validation is more robust (no “easy examples”)
![Page 14: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/14.jpg)
Supervised Data Mining: The Line Drawing ContestWhy are lines a big deal again?
The actual underlying hypothesis is unknown
Lots of features in our dataset makes it difficult to draw them by hand
Simulating intelligent behaviour has complex lines
AlphaGo was just a very complex model which predicted the next move to make
![Page 15: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/15.jpg)
Supervised Data Mining: The Line Drawing ContestWhy are lines a big deal again?
The actual underlying hypothesis is unknown
![Page 16: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/16.jpg)
Supervised Data Mining: The Line Drawing Contest
Why are lines a big deal again?
The actual underlying hypothesis is unknown
Infinitely many ways we can create lines
Lots of features in our dataset makes it difficult to draw them by hand
![Page 17: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/17.jpg)
Supervised Data Mining: The Line Drawing Contest
Simulating truly intelligent behaviour has complex lines/curves
Keep this in mind:
AlphaGo was just a very complex predictive model which predicted the next move to make in Go
It took them months with a supercomputer to build this model
![Page 18: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/18.jpg)
Supervised Data Mining: The Line Drawing ContestWho can draw the most “realistic” line?
K-nearest Neighbours
Support Vector Machines
Grid
K = 5 to 10
Grid
C = 0.001 to 10
Hei
ght
Width
Hei
ght
Width
Data (in a table form) Decision
Trees (“rpart”)
Grid
cp = 0.01 to 0.10
Hei
ght
Width
Best K: 35-CV Accuracy: 95%
Best cp: 0.025-CV Accuracy: 98%
Best C: 95-CV Accuracy: 100%
![Page 19: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/19.jpg)
Support Vector Machines: How do they draw lines?They are another supervised data mining technique used for either regression or classification
Invented by Vladimir Vapnik (now at Facebook)
![Page 20: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/20.jpg)
Support Vector Machines: How do they draw lines?Let’s look at two-class classification first.
Consider the classification scenario below:
How could we create a classifier which best divides the two classes?
![Page 21: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/21.jpg)
Support Vector Machines: How do they draw lines?Let’s look at classification first. Consider the classification scenario below:
Hmm - probably right there.
![Page 22: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/22.jpg)
Support Vector Machines: How do they draw lines?It’s “right in between” both classes and divides them both pretty well.
It is a line equidistant from the “outside” points of each class
![Page 23: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/23.jpg)
Support Vector Machines: How do they draw lines?There are two steps for this:
1. Identify these “outside” points
2. Draw a line equidistant between both sets of outside points
![Page 24: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/24.jpg)
Support Vector Machines: How do they draw lines?Step 1: Identify these “outside” points.
They are called the “support vectors” in the SVM model.
![Page 25: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/25.jpg)
Support Vector Machines: How do they draw lines?Step 2: Draw a line equidistant between support vectors
Draw the dividing line which is perpendicular to the margin with the furthest distance between the boundaries of the support vectors
![Page 26: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/26.jpg)
Support Vector Machines: How do they draw lines?Step 2: Draw a line equidistant between support vectors
![Page 27: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/27.jpg)
Support Vector Machines: How do they draw lines?Step 2: Draw a line equidistant between both classes
Final Exam: I’ll ask a question related to how/why does the SVM draw this line.
![Page 28: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/28.jpg)
Step 2: Draw a line equidistant between both classes
To find this 'middle-ground' line
Consider that we need to find the appropriate slope and intercept of the line with respect to an optimization task of maximizing the margin between support vectors.
![Page 29: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/29.jpg)
Step 2: Draw a line equidistant between both classes
The idea is easy to understand, but there is beautiful math behind the scenes to find this line.
If you are interested in this, please have a look at Lecture Notes.
![Page 30: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/30.jpg)
Support Vector Machines: How do they draw lines?In a nutshell, a predictive model built with SVM has two steps:
Step 1: Find the “outside” data points (called the support vectors)
Step 2: Draw line equidistant between them.
Step 1: Find Support Vectors Step 2: Draw Equidistant Line
![Page 31: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/31.jpg)
Support Vector Machines: Some issues
What about noisy observations?
Deformed cats may ruin the equidistant line
What if noise were chosen as a support vector?
![Page 32: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/32.jpg)
Noisy Observations can ruin the lineIf the support vectors were noise, we may get something like this...
![Page 33: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/33.jpg)
Noisy Observations can ruin the line
A single noisy example messes up everything
The support vector is incorrectly chosen.
Our model is invalid and has overfit
It wouldn’t generalize very well to real-world cases
![Page 34: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/34.jpg)
Support Vector Machines: The Cost Parameter
Like K in K-nearest Neighbours
Like “cp” and “max depth” in Decision Trees
We have a hyperparameter to control complexity of the model
That is, how tolerable it is to noisy observations in our data
![Page 35: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/35.jpg)
Support Vector Machines: The Cost Parameter
We can specify the “Cost” hyperparameter or “C” to avoid noise.
A way of determining the resistance of the chosen support vectors to noise.
![Page 36: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/36.jpg)
Support Vector Machines: The Cost Parameter
C = 1,000 C = 0.001
![Page 37: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/37.jpg)
Support Vector Machines: The Cost Parameter
Cost (C): When picking the support vectors, the “cost” of incorrectly classifying a data point.
![Page 38: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/38.jpg)
Support Vector Machines: The Cost Parameter
Cost (C): When picking the support vectors, the “cost” of incorrectly classifying a data point.
Higher C values means that there is a higher cost to incorrectly classifying a training point. Too high means we’ll underfit.
Lower C values means that there is a lower cost to incorrectly classifying a training point. Too low means we’ll overfit.
![Page 39: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/39.jpg)
Support Vector Machines: The Cost Parameter
The cost is high to make mistakes on the training data.Since the cost is high, we can’t make mistakesLet’s draw a line here
C = 1,000
![Page 40: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/40.jpg)
Support Vector Machines: The Cost Parameter
The cost is low to make mistakes on the training data.Since the cost is low, we can make some mistakesLet’s draw a line here
C = 0.001
![Page 41: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/41.jpg)
Support Vector Machines: The Cost Parameter
“Cost” for SVMs is like K for KNN (or cp for Decision Trees)
We just try a bunch of different cost values until we find a good one
The one that will develop a model that works well.
Find one that avoids both overfitting and underfitting.
![Page 42: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/42.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines
Multi-class Support Vector Machines ←
Kernels and Support Vector Machines
Support Vector Machines for Regression
![Page 43: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/43.jpg)
Support Vector Machines: Multi-class Problems
What if we had more than two classes?
How would we draw the line now?
Hei
ght
Width
![Page 44: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/44.jpg)
Support Vector Machines: Multi-class Problems
K-Nearest Neighbours and Decision Trees are naturally made to handle multi-class tasks
SVMs are not made for classification tasks with multiple classes.
![Page 45: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/45.jpg)
The “One-vs-One” trick for Multi-Class SVMs
We can’t use SVMs by themselves for multi-class problems
We can use a trick for SVMs to solve multiclass problems
![Page 46: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/46.jpg)
The “One-vs-One” trick for Multi-Class SVMs
We train three classifiers for all combination of classes:
● Cat vs. Parrot● Cat vs. Mouse● Mouse vs. Parrot
Run SVM three different times
When an unknown observation comes in, we evaluate the point with each classifier.
Do a majority vote as final prediction.
![Page 47: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/47.jpg)
The “One-vs-One” trick for Multi-Class SVMsH
eigh
t
Width
Hei
ght
Width
Cat vs Parrot
Cat vs Mouse
Hei
ght
Width
Hei
ght
Width
Parrot vs Mouse
Original Data
We must create three separate SVM models
![Page 48: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/48.jpg)
The “One-vs-One” trick for Multi-Class SVMs
Our predictive model is composed of sub-models.
Three different SVM sub-models
Predictive Model
![Page 49: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/49.jpg)
The “One-vs-One” trick for Multi-Class SVMsPredictive Model
What Species is this?
<4.2, 5.4>
Cat
Mouse
Mouse
![Page 50: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/50.jpg)
The “One-vs-One” trick for Multi-Class SVMsPredictive Model
What Species is this?
<4.2, 5.4>
Cat
Mouse
MouseMouse
![Page 51: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/51.jpg)
The “One-vs-One” trick: Summary
This is sort-of like cheating
But SVMs cannot handle multi-classes by themselves
![Page 52: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/52.jpg)
The “One-vs-One” trick: Summary
SVM uses the One-vs-One trick for multi-class problems
Sub-models are built each possible class combination
Majority Vote afterwards for final prediction
R does this trick for us in the background
![Page 53: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/53.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines
Multi-class Support Vector Machines
Kernels and Support Vector Machines ←
Support Vector Machines for Regression
![Page 54: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/54.jpg)
Support Vector Machines
A support vector machine can find a linear decision boundary between two classes
But what if the underlying function of our data is non-linear?
I.e. It is a curvy decision surface?
Hei
ght
Width
![Page 55: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/55.jpg)
Support Vector Machines
The raw SVM does a terrible job here.
The underlying data cannot be linearly separated
Hei
ght
Width
Hei
ght
Width
SVM
![Page 56: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/56.jpg)
Pre-processing: Non-Linear Kernels
To use the support vector classifier with non-linear data, there is only one twist to what we have seen earlier.
We have to “pre-process” the given data with a non-linear transformation function.
Hopefully after this transformation, the regular SVM will work properly.
![Page 57: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/57.jpg)
Non-Linear Support Vector Machines
There are four steps now.
1. Pre-process the data with a non-linear function2. Identify support vectors
3. Draw a line equidistant between both classes
4. Project the line back onto original space
![Page 58: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/58.jpg)
Non-Linear Support Vector Machines
Our original data not linearly separable.
Using a plain SVM here would give us a predictor with terrible performance.
![Page 59: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/59.jpg)
Non-Linear Support Vector Machines
Transform the given data using a kernel function.
We hope that after applying a non-linear kernel to the data, we can apply a regular SVM and get good accuracy
![Page 60: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/60.jpg)
Non-Linear Support Vector MachinesThe support vector classifier is applied in transformed feature space
The line is drawn separating the two classes apart
Step 2 and Step 3
Apply the regular SVM in this transformed space.
Find the “middle” ground line
![Page 61: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/61.jpg)
Non-Linear Support Vector MachinesStep 4: Projecting the decision surface back onto our original feature space
We get a non-linear decision boundary
Project Back
![Page 62: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/62.jpg)
Non-Linear Support Vector MachinesThis is an awesome video that gives better intuition on how kernels work.
Video: https://www.youtube.com/watch?v=3liCbRZPrZA
![Page 63: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/63.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines
Multi-class Support Vector Machines
Kernels and Support Vector Machines
Support Vector Machines for Regression ←
![Page 64: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/64.jpg)
Support Vector Machines for Regression
We talked about SVM for classification
Building predictive models that predict categories
What about for regression?
Building predictive models that predict numbers
![Page 65: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/65.jpg)
Support Vector Machines for Regression
Basically the same procedure.
Step 1. Identify Support Vectors
Step 2. Draw “Middle” Line
But now the support vectors are “outside points” of our entire data
* Maybe a few more steps if using a kernel transformation
![Page 66: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/66.jpg)
Support Vector Machines for Regression
This is our data set that we are given.
![Page 67: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/67.jpg)
Support Vector Machines for Regression
This is our data set that we are given.
![Page 68: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/68.jpg)
Support Vector Machines for Regression
Step 1: Identify Support Vectors (Outside Points)
![Page 69: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/69.jpg)
Support Vector Machines for Regression
Step 1: Identify Support Vectors (Outside Points)
![Page 70: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/70.jpg)
Support Vector Machines for Regression
Step 2: Draw line equidistant from support vectors
![Page 71: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/71.jpg)
Support Vector Machines: For Regression
![Page 72: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/72.jpg)
Outline for Today
Support Vector Machines - Another way to draw lines
Multi-class Support Vector Machines
Kernels and Support Vector Machines
Support Vector Machines for Regression
![Page 73: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/73.jpg)
Assignment 2 Review
Any questions/confusions/worries?
Lemme know! =)
Link: http://web.cs.dal.ca/~kallada/stat2450/assignments/Assignment2.pdf
![Page 74: STAT2450 - Introduction to Data Mining with R Vector Machineskallada/stat2450/lectures/Lecture10_Slides.pdf · Supervised Data Mining: The Line Drawing Contest To evaluate whether](https://reader033.fdocuments.us/reader033/viewer/2022042419/5f366e8c9627065158430af1/html5/thumbnails/74.jpg)
That’s all for today
Assignment 2 is due next Tuesday!
I will be away next Monday.