Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier...
Transcript of Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier...
![Page 1: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/1.jpg)
Image classification
16-385 Computer VisionSpring 2018, Lecture 18http://www.cs.cmu.edu/~16385/
![Page 2: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/2.jpg)
• Homework 5 has been posted and is due on April 6th.- Dropbox link because course website is out of space…- Any questions about the homework?- How many of you have looked at/started/finished homework 5?
• Yannis’ office hours this Friday will be 3 – 4:30pm instead of 3 – 5pm.- I’ll make it up by scheduling additional office hours next week on Wednesday.
• How many of you went to Saurabh Gupta’s talk yesterday?
• Talk: Matthew O’Toole, “Probing light transport for 3D shape,” Thursday, 1:00 PM, GHC 6115.
Course announcements
![Page 3: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/3.jpg)
Epipolar imaging camera
Video stream of direct reflections Video stream of indirect reflections
![Page 4: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/4.jpg)
![Page 5: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/5.jpg)
![Page 6: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/6.jpg)
![Page 7: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/7.jpg)
![Page 8: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/8.jpg)
cameras that see around corners
cameras that measure depth in real time
15-463/15-663/15-862 Computational Photography
Learn about this and other unconventional cameras – and build some on your own!
http://graphics.cs.cmu.edu/courses/15-463/
cameras that take video at the speed of light
cameras that capture entire focal stacks
Matt O’Toole will talk about all of these tomorrow
![Page 9: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/9.jpg)
• Bag-of-words.
• K-means clustering.
• Classification.
• K nearest neighbors.
• Naïve Bayes.
• Support vector machine.
Overview of today’s lecture
![Page 10: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/10.jpg)
Slide credits
Most of these slides were adapted from:
• Kris Kitani (16-385, Spring 2017).
• Noah Snavely (Cornell University).
• Fei-Fei Li (Stanford University).
![Page 11: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/11.jpg)
Image Classification
![Page 12: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/12.jpg)
Image Classification: Problem
![Page 13: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/13.jpg)
Data-driven approach
• Collect a database of images with labels
• Use ML to train an image classifier
• Evaluate the classifier on test images
![Page 14: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/14.jpg)
Bag of words
![Page 15: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/15.jpg)
What object do these parts belong to?
![Page 16: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/16.jpg)
a collection of local features(bag-of-features)
An object as
Some local feature are
very informative
• deals well with occlusion
• scale invariant
• rotation invariant
![Page 17: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/17.jpg)
(not so) crazy assumption
spatial information of local features
can be ignored for object recognition (i.e., verification)
![Page 18: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/18.jpg)
Csurka et al. (2004), Willamowski et al. (2005), Grauman & Darrell (2005), Sivic et al. (2003, 2005)
Works pretty well for image-level classification
CalTech6 dataset
![Page 19: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/19.jpg)
Bag-of-features
an old idea(e.g., texture recognition and information retrieval)
represent a data item (document, texture, image)
as a histogram over features
![Page 20: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/20.jpg)
Texture recognition
Universal texton dictionary
histogram
![Page 21: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/21.jpg)
Vector Space ModelG. Salton. ‘Mathematics and Information Retrieval’ Journal of Documentation,1979
1 6 2 1 0 0 0 1
Tartan robot CHIMP CMU bio soft ankle sensor
0 4 0 1 4 5 3 2
Tartan robot CHIMP CMU bio soft ankle sensor
http://www.fodey.com/generators/newspaper/snippet.asp
![Page 22: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/22.jpg)
A document (datapoint) is a vector of counts over each word (feature)
What is the similarity between two documents?
counts the number of occurrences just a histogram over words
![Page 23: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/23.jpg)
A document (datapoint) is a vector of counts over each word (feature)
What is the similarity between two documents?
counts the number of occurrences just a histogram over words
Use any distance you want but the cosine distance is fast.
![Page 24: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/24.jpg)
but not all words are created equal
![Page 25: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/25.jpg)
TF-IDF
weigh each word by a heuristic
Term Frequency Inverse Document Frequency
term
frequency
inverse document
frequency
(down-weights common terms)
![Page 26: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/26.jpg)
Standard BOW pipeline(for image classification)
![Page 27: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/27.jpg)
Dictionary Learning:
Learn Visual Words using clustering
Encode:
build Bags-of-Words (BOW) vectors
for each image
Classify:
Train and test data using BOWs
![Page 28: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/28.jpg)
Dictionary Learning:
Learn Visual Words using clustering
1. extract features (e.g., SIFT) from images
![Page 29: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/29.jpg)
Dictionary Learning:
Learn Visual Words using clustering
2. Learn visual dictionary (e.g., K-means clustering)
![Page 30: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/30.jpg)
What kinds of features can we extract?
![Page 31: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/31.jpg)
• Regular grid• Vogel & Schiele, 2003
• Fei-Fei & Perona, 2005
• Interest point detector• Csurka et al. 2004
• Fei-Fei & Perona, 2005
• Sivic et al. 2005
• Other methods• Random sampling (Vidal-Naquet &
Ullman, 2002)
• Segmentation-based patches (Barnard et al. 2003)
![Page 32: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/32.jpg)
Normalize patch
Detect patches[Mikojaczyk and Schmid ’02]
[Mata, Chum, Urban & Pajdla, ’02]
[Sivic & Zisserman, ’03]
Compute SIFT descriptor
[Lowe’99]
![Page 33: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/33.jpg)
…
![Page 34: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/34.jpg)
How do we learn the dictionary?
![Page 35: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/35.jpg)
…
![Page 36: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/36.jpg)
Clustering
…
![Page 37: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/37.jpg)
Clustering
…Visual vocabulary
![Page 38: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/38.jpg)
K-means clustering
![Page 39: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/39.jpg)
1. Select initial
centroids at random
![Page 40: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/40.jpg)
1. Select initial
centroids at random
2. Assign each object to
the cluster with the
nearest centroid.
![Page 41: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/41.jpg)
1. Select initial
centroids at random
2. Assign each object to
the cluster with the
nearest centroid.
3. Compute each centroid as the
mean of the objects assigned to
it (go to 2)
![Page 42: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/42.jpg)
1. Select initial
centroids at random
2. Assign each object to
the cluster with the
nearest centroid.
3. Compute each centroid as the
mean of the objects assigned to
it (go to 2)
2. Assign each object to
the cluster with the
nearest centroid.
![Page 43: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/43.jpg)
1. Select initial
centroids at random
2. Assign each object to
the cluster with the
nearest centroid.
3. Compute each centroid as the
mean of the objects assigned to
it (go to 2)
2. Assign each object to
the cluster with the
nearest centroid.
Repeat previous 2 steps until no change
![Page 44: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/44.jpg)
K-means Clustering
Given k:
1.Select initial centroids at random.
2.Assign each object to the cluster with the nearest
centroid.
3.Compute each centroid as the mean of the objects
assigned to it.
4.Repeat previous 2 steps until no change.
![Page 45: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/45.jpg)
From what data should I learn the dictionary?
• Codebook can be learned on separate training set
• Provided the training set is sufficiently
representative, the codebook will be “universal”
![Page 46: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/46.jpg)
From what data should I learn the dictionary?
• Dictionary can be learned on separate training set
• Provided the training set is sufficiently
representative, the dictionary will be “universal”
![Page 47: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/47.jpg)
Example visual dictionary
![Page 48: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/48.jpg)
Example dictionary
…
Source: B. Leibe
Appearance codebook
![Page 49: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/49.jpg)
Another dictionary
Appearance codebook…
…
……
…
Source: B. Leibe
![Page 50: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/50.jpg)
Dictionary Learning:
Learn Visual Words using clustering
Encode:
build Bags-of-Words (BOW) vectors
for each image
Classify:
Train and test data using BOWs
![Page 51: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/51.jpg)
Encode:
build Bags-of-Words (BOW) vectors
for each image
1. Quantization: image features gets
associated to a visual word (nearest
cluster center)
![Page 52: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/52.jpg)
Encode:
build Bags-of-Words (BOW) vectors
for each image 2. Histogram: count the
number of visual word
occurrences
![Page 53: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/53.jpg)
…..
freq
uen
cy
codewords
![Page 54: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/54.jpg)
Dictionary Learning:
Learn Visual Words using clustering
Encode:
build Bags-of-Words (BOW) vectors
for each image
Classify:
Train and test data using BOWs
![Page 55: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/55.jpg)
K nearest neighbors
Naïve Bayes
Support Vector Machine
![Page 56: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/56.jpg)
K nearest neighbors
![Page 57: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/57.jpg)
Distribution of data from two classes
![Page 58: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/58.jpg)
Which class does q belong too?
Distribution of data from two classes
![Page 59: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/59.jpg)
Distribution of data from two classes
Look at the neighbors
![Page 60: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/60.jpg)
K-Nearest Neighbor (KNN) Classifier
Non-parametric pattern classification
approach
Consider a two class problem where
each sample consists of two
measurements (x,y).
k = 1
k = 3
For a given query point q,
assign the class of the nearest
neighbor
Compute the k nearest
neighbors and assign the
class by majority vote.
![Page 61: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/61.jpg)
Nearest Neighbor is competitive
MNIST Digit Recognition
– Handwritten digits
– 28x28 pixel images: d = 784
– 60,000 training samples
– 10,000 test samples
Test Error Rate (%)
Linear classifier (1-layer NN) 12.0
K-nearest-neighbors, Euclidean 5.0
K-nearest-neighbors, Euclidean, deskewed 2.4
K-NN, Tangent Distance, 16x16 1.1
K-NN, shape context matching 0.67
1000 RBF + linear classifier 3.6
SVM deg 4 polynomial 1.1
2-layer NN, 300 hidden units 4.7
2-layer NN, 300 HU, [deskewing] 1.6
LeNet-5, [distortions] 0.8
Boosted LeNet-4, [distortions] 0.7Yann LeCunn
![Page 62: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/62.jpg)
What is the best distance metric between data points?
• Typically Euclidean distance
• Locality sensitive distance metrics
• Important to normalize. Dimensions have different scales
How many K?
• Typically k=1 is good
• Cross-validation (try different k!)
![Page 63: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/63.jpg)
Distance metrics
Cosine
Chi-squared
Euclidean
![Page 64: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/64.jpg)
Choice of distance metric
• Hyperparameter
![Page 65: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/65.jpg)
Visualization: L2 distance
![Page 66: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/66.jpg)
CIFAR-10 and NN results
![Page 67: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/67.jpg)
k-nearest neighbor
• Find the k closest points from training data
• Labels of the k points “vote” to classify
![Page 68: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/68.jpg)
Hyperparameters
• What is the best distance to use?
• What is the best value of k to use?
• i.e., how do we set the hyperparameters?
• Very problem-dependent
• Must try them all and see what works best
![Page 69: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/69.jpg)
![Page 70: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/70.jpg)
![Page 71: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/71.jpg)
Validation
![Page 72: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/72.jpg)
Cross-validation
![Page 73: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/73.jpg)
![Page 74: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/74.jpg)
How to pick hyperparameters?
• Methodology
– Train and test
– Train, validate, test
• Train for original model
• Validate to find hyperparameters
• Test to understand generalizability
![Page 75: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/75.jpg)
Pros
• simple yet effective
Cons
• search is expensive (can be sped-up)
• storage requirements
• difficulties with high-dimensional data
![Page 76: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/76.jpg)
kNN -- Complexity and Storage
• N training images, M test images
• Training: O(1)
• Testing: O(MN)
• Hmm…
– Normally need the opposite
– Slow training (ok), fast testing (necessary)
![Page 77: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/77.jpg)
![Page 78: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/78.jpg)
Naïve Bayes
![Page 79: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/79.jpg)
Which class does q belong too?
Distribution of data from two classes
![Page 80: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/80.jpg)
Distribution of data from two classes
• Learn parametric model for each class
• Compute probability of query
![Page 81: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/81.jpg)
This is called the posterior.
the probability of a class z given the observed features X
For classification, z is a
discrete random variable(e.g., car, person, building)
X is a set of observed features(e.g., features from a single image)
(it’s a function that returns a single probability value)
![Page 82: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/82.jpg)
Each x is an observed feature (e.g., visual words)
(it’s a function that returns a single probability value)
This is called the posterior:
the probability of a class z given the observed features X
For classification, z is a
discrete random variable(e.g., car, person, building)
![Page 83: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/83.jpg)
posterior
likelihood prior
The posterior can be decomposed according to
Bayes’ Rule
In our context…
Recall:
![Page 84: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/84.jpg)
The naive Bayes’ classifier is solving this optimization
MAP (maximum a posteriori) estimate
Bayes’ Rule
Remove constants
To optimize this…we need to compute this
Compute the likelihood…
![Page 85: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/85.jpg)
A naive Bayes’ classifier assumes all features are
conditionally independent
Recall:
![Page 86: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/86.jpg)
To compute the MAP estimate
Given (1) a set of known parameters
Compute which z has the largest probability
(2) observations
![Page 87: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/87.jpg)
count 1 6 2 1 0 0 0 1
word Tartan robot CHIMP CMU bio soft ankle sensor
p(x|z) 0.09 0.55 0.18 0.09 0.0 0.0 0.0 0.09
* typically add pseudo-counts (0.001)
** this is an example for computing the likelihood, need to multiply times prior to get posterior
Numbers get really small so use log probabilities
![Page 88: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/88.jpg)
count 1 6 2 1 0 0 0 1
word Tartan robot CHIMP CMU bio soft ankle sensor
p(x|z) 0.09 0.55 0.18 0.09 0.0 0.0 0.0 0.09
count 0 4 0 1 4 5 3 2
word Tartan robot CHIMP CMU bio soft ankle sensor
p(x|z) 0.0 0.21 0.0 0.05 0.21 0.26 0.16 0.11http://www.fodey.com/generators/newspaper/snippet.asp
log p(X|z=grand challenge) = - 14.58
log p(X|z=bio inspired) = - 37.48
log p(X|z=grand challenge) = - 94.06
log p(X|z=bio inspired) = - 32.41
* typically add pseudo-counts (0.001)
** this is an example for computing the likelihood, need to multiply times prior to get posterior
![Page 89: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/89.jpg)
Support Vector Machine
![Page 90: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/90.jpg)
Image Classification
![Page 91: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/91.jpg)
Score function
![Page 92: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/92.jpg)
Linear Classifier
data (histogram)
![Page 93: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/93.jpg)
Convert image to histogram representation
![Page 94: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/94.jpg)
Which class does q belong too?
Distribution of data from two classes
![Page 95: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/95.jpg)
Distribution of data from two classes
Learn the decision boundary
![Page 96: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/96.jpg)
First we need to understand hyperplanes…
![Page 97: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/97.jpg)
Hyperplanes (lines) in 2D
a line can be written as
dot product plus a bias
another version, add a weight 1 and
push the bias inside
![Page 98: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/98.jpg)
Hyperplanes (lines) in 2D
(offset/bias outside) (offset/bias inside)
![Page 99: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/99.jpg)
define the same line
The line
and the line
Important property:
Free to choose any normalization of w
Hyperplanes (lines) in 2D
(offset/bias outside) (offset/bias inside)
![Page 100: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/100.jpg)
What is the distance
to origin?
(hint: use normal form)
![Page 101: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/101.jpg)
you get the normal form
distance to origin
scale by
![Page 102: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/102.jpg)
What is the distance
between two parallel lines?(hint: use distance to origin)
![Page 103: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/103.jpg)
distance
between two
parallel lines
Difference of distance to origin
![Page 104: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/104.jpg)
What happens if you change b?
Hyperplanes (planes) in 3D
Now we can go to 3D …
what are the dimensions of
this vector?
![Page 105: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/105.jpg)
Hyperplanes (planes) in 3D
![Page 106: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/106.jpg)
What’s the distance
between these
parallel planes?
Hyperplanes (planes) in 3D
![Page 107: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/107.jpg)
Hyperplanes (planes) in 3D
![Page 108: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/108.jpg)
What’s the best w?
![Page 109: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/109.jpg)
What’s the best w?
![Page 110: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/110.jpg)
What’s the best w?
![Page 111: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/111.jpg)
What’s the best w?
![Page 112: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/112.jpg)
What’s the best w?
![Page 113: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/113.jpg)
What’s the best w?
Intuitively, the line that is the
farthest from all interior points
![Page 114: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/114.jpg)
What’s the best w?
Maximum Margin solution:
most stable to perturbations of data
![Page 115: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/115.jpg)
What’s the best w?
Want a hyperplane that is far away from ‘inner points’
support vectors
![Page 116: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/116.jpg)
Find hyperplane w such that …
the gap between parallel hyperplanes
margin
is maximized
![Page 117: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/117.jpg)
Can be formulated as a maximization problem
label of the data point
Why is it +1 and -1?
What does this constraint mean?
![Page 118: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/118.jpg)
Can be formulated as a maximization problem
Equivalently,Where did the 2 go?
What happened to the labels?
![Page 119: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/119.jpg)
Objective Function
Constraints
‘Primal formulation’ of a linear SVM
This is a convex quadratic programming (QP) problem
(a unique solution exists)
![Page 120: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/120.jpg)
‘soft’ margin
![Page 121: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/121.jpg)
What’s the best w?
![Page 122: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/122.jpg)
What’s the best w?
Very narrow margin
![Page 123: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/123.jpg)
Separating cats and dogs
Very narrow margin
![Page 124: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/124.jpg)
What’s the best w?
Very narrow margin
Intuitively, we should allow for some misclassification if we
can get more robust classification
![Page 125: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/125.jpg)
What’s the best w?
Trade-off between the MARGIN and the MISTAKES
(might be a better solution)
![Page 126: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/126.jpg)
Adding slack variables
misclassified
point
![Page 127: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/127.jpg)
‘soft’ margin
objective subject to
for
![Page 128: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/128.jpg)
‘soft’ margin
objective subject to
for
The slack variable allows for mistakes,
as long as the inverse margin is minimized.
![Page 129: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/129.jpg)
‘soft’ margin
subject to
for
objective
• Every constraint can be satisfied if slack is large
• C is a regularization parameter
• Small C: ignore constraints (larger margin)
• Big C: constraints (small margin)
• Still QP problem (unique solution)
![Page 130: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/130.jpg)
![Page 131: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/131.jpg)
![Page 132: Image classification16385/s18/lectures/lecture18.pdfK-Nearest Neighbor (KNN) Classifier Non-parametric pattern classification approach Consider a two class problem where each sample](https://reader030.fdocuments.us/reader030/viewer/2022041016/5ec774ebe0ecc52c352cf114/html5/thumbnails/132.jpg)
References
Basic reading:• Szeliski, Chapter 14.