Lecture: Face Recognition
Transcript of Lecture: Face Recognition
![Page 1: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/1.jpg)
Face identification
Stanford University
05-Nov-2019
1
Lecture: Face Recognition
Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab
![Page 2: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/2.jpg)
Face identification
Stanford University
05-Nov-2019
2
CS 131 Roadmap
Pixels Images
ConvolutionsEdgesDescriptors
Segments
ResizingSegmentationClustering
RecognitionDetectionMachine learning
Videos
MotionTracking
Web
Neural networksConvolutional neural networks
![Page 3: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/3.jpg)
Face identification
Stanford University
05-Nov-2019
3
Let’s recap
• A simple object recognition pipeline with kNN• PCA
![Page 4: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/4.jpg)
Face identification
Stanford University
05-Nov-2019
4
Object recognition: a classification framework
• Apply a prediction function to a feature representation of the image to get the desired output:
f( ) = “apple”f( ) = “tomato”f( ) = “cow”
Slide credit: L. LazebnikDataset: ETH-80, by B. Leibe
![Page 5: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/5.jpg)
Face identification
Stanford University
05-Nov-2019
5
A simple pipeline - Training
Training Images
Image Features
![Page 6: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/6.jpg)
Face identification
Stanford University
05-Nov-2019
6
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
![Page 7: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/7.jpg)
Face identification
Stanford University
05-Nov-2019
7
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Learned Classifier
![Page 8: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/8.jpg)
Face identification
Stanford University
05-Nov-2019
8
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
![Page 9: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/9.jpg)
Face identification
Stanford University
05-Nov-2019
9
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
![Page 10: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/10.jpg)
Face identification
Stanford University
05-Nov-2019
10
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
![Page 11: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/11.jpg)
Face identification
Stanford University
05-Nov-2019
11
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
![Page 12: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/12.jpg)
Face identification
Stanford University
05-Nov-2019
12
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 13: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/13.jpg)
Face identification
Stanford University
05-Nov-2019
13
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 14: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/14.jpg)
Face identification
Stanford University
05-Nov-2019
14
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 15: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/15.jpg)
Face identification
Stanford University
05-Nov-2019
15
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 16: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/16.jpg)
Face identification
Stanford University
05-Nov-2019
16
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 17: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/17.jpg)
Face identification
Stanford University
05-Nov-2019
17
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 18: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/18.jpg)
Face identification
Stanford University
05-Nov-2019
18
Image featuresInput image
Color: Quantize RGB values Invariance?? Translation? Scale? Rotation? Occlusion
Global shape: PCA space Invariance?? Translation? Scale? Rotation? Occlusion
Local shape: shape context Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
Texture: Filter banks Invariance?? Translation? Scale? Rotation (in-planar)
? Occlusion
![Page 19: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/19.jpg)
Face identification
Stanford University
05-Nov-2019
19
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
![Page 20: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/20.jpg)
Face identification
Stanford University
05-Nov-2019
20
Classifiers: Nearest neighbor
Training examples
from class 1
Training examples
from class 2
Slide credit: L. Lazebnik
![Page 21: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/21.jpg)
Face identification
Stanford University
05-Nov-2019
21
Prediction
A simple pipeline - TrainingTraining LabelsTraining
Images
TrainingImage Features
Image Features
Test Image
Learned Classifier
Learned Classifier
![Page 22: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/22.jpg)
Face identification
Stanford University
05-Nov-2019
22
Classifiers: Nearest neighbor
Test example
Training examples
from class 1
Training examples
from class 2
Slide credit: L. Lazebnik
![Page 23: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/23.jpg)
Face identification
Stanford University
05-Nov-2019
23
Let’s recap
• A simple object recognition pipeline with kNN• PCA
![Page 24: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/24.jpg)
Face identification
Stanford University
05-Nov-2019
24
PCA compression: 144D -> 6D
![Page 25: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/25.jpg)
Face identification
Stanford University
05-Nov-2019
25
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
2 4 6 8 10 12
24681012
6 most important eigenvectors
![Page 26: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/26.jpg)
Face identification
Stanford University
05-Nov-2019
26
PCA compression: 144D ) 3D
![Page 27: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/27.jpg)
Face identification
Stanford University
05-Nov-2019
27
2 4 6 8 10 12
2
4
6
8
10
122 4 6 8 10 12
2
4
6
8
10
12
2 4 6 8 10 12
2
4
6
8
10
12
3 most important eigenvectors
![Page 28: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/28.jpg)
Face identification
Stanford University
05-Nov-2019
28
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
![Page 29: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/29.jpg)
Face identification
Stanford University
05-Nov-2019
29Courtesy of Johannes M. Zanker
“Faces” in the brain
![Page 30: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/30.jpg)
Face identification
Stanford University
05-Nov-2019
30
“Faces” in the brain fusiform face area
Kanwisher, et al. 1997
![Page 31: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/31.jpg)
Face identification
Stanford University
05-Nov-2019
31
Detection versus Recognition
Detection finds the faces in images Recognition recognizes WHO the person is
![Page 32: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/32.jpg)
Face identification
Stanford University
05-Nov-2019
32
Face Recognition
• Digital photography
![Page 33: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/33.jpg)
Face identification
Stanford University
05-Nov-2019
33
Face Recognition
• Digital photography• Surveillance
![Page 34: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/34.jpg)
Face identification
Stanford University
05-Nov-2019
34
Face Recognition
• Digital photography• Surveillance• Album organization
![Page 35: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/35.jpg)
Face identification
Stanford University
05-Nov-2019
35
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.
![Page 36: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/36.jpg)
Face identification
Stanford University
05-Nov-2019
36
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions
![Page 37: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/37.jpg)
Face identification
Stanford University
05-Nov-2019
37
Face Recognition
• Digital photography• Surveillance• Album organization• Person tracking/id.• Emotions and expressions• Security/warfare• Tele-conferencing• Etc.
![Page 38: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/38.jpg)
Face identification
Stanford University
05-Nov-2019
38
The Space of Faces
• An image is a point in a high dimensional space– If represented in grayscale intensity,
an N x M image is a point in RNM
– E.g. 100x100 image = 10,000 dim
Slide credit: Chuck Dyer, Steve Seitz, Nishino
![Page 39: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/39.jpg)
Face identification
Stanford University
05-Nov-2019
39
100x100 images can contain many things other than faces!
![Page 40: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/40.jpg)
Face identification
Stanford University
05-Nov-2019
40
The Space of Faces
• An image is a point in a high dimensional space– If represented in grayscale intensity,
an N x M image is a point in RNM
– E.g. 100x100 image = 10,000 dim
• However, relatively few high dimensional vectors correspond to valid face images
• We want to effectively model the subspace of face images
Slide credit: Chuck Dyer, Steve Seitz, Nishino
ɸ1
![Page 41: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/41.jpg)
Face identification
Stanford University
05-Nov-2019
41
Where have we seen something like this before?
ɸ1
![Page 42: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/42.jpg)
Face identification
Stanford University
05-Nov-2019
42
Image space Face space
• Maximize the scatter of the training images in face space
• Compute n-dim subspace such that the projection of the data points onto the subspace has the largest variance among all n-dim subspaces.
![Page 43: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/43.jpg)
Face identification
Stanford University
05-Nov-2019
43
• So, compress them to a low-dimensional subspace thatcaptures key appearance characteristics of the visual DOFs.
Key Idea
• USE PCA for estimating the sub-space (dimensionality reduction)
•Compare two faces by projecting the images into the subspace and measuring the EUCLIDEAN distance between them.
![Page 44: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/44.jpg)
Face identification
Stanford University
05-Nov-2019
44
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
![Page 45: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/45.jpg)
Face identification
Stanford University
05-Nov-2019
45
Eigenfaces: key idea
• Assume that most face images lie on a low-dimensional subspace determined by the first k (k<<d) directions of maximum variance
• Use PCA to determine the vectors or “eigenfaces” that span that subspace• Represent all face images in the dataset as linear combinations of eigenfaces
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
![Page 46: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/46.jpg)
Face identification
Stanford University
05-Nov-2019
46
Training images: x1,…,xN
![Page 47: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/47.jpg)
Face identification
Stanford University
05-Nov-2019
47
Top eigenvectors: ɸ1,…,ɸk
Mean: μ
![Page 48: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/48.jpg)
Face identification
Stanford University
05-Nov-2019
48
Visualization of eigenfacesPrincipal component (eigenvector) ɸk
μ + 3σkɸk
μ – 3σkɸk
![Page 49: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/49.jpg)
Face identification
Stanford University
05-Nov-2019
49
Eigenface algorithm
• Training1. Align training images x1, x2, …, xN
2. Compute average face
3. Compute the difference image (the centered data matrix)
Note that each image is formulated into a long vector!
µ =1N
xi∑
![Page 50: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/50.jpg)
Face identification
Stanford University
05-Nov-2019
50
Eigenface algorithm
4. Compute the covariance matrix
5. Compute the eigenvectors of the covariance matrix Σ6. Compute each training image xi ‘s projections as
7. Visualize the estimated training face xi
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
![Page 51: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/51.jpg)
Face identification
Stanford University
05-Nov-2019
51
Eigenface algorithm
6. Compute each training image xi ‘s projections as
7. Visualize the reconstructed training face xi
xi → xic ⋅φ1, xi
c ⋅φ2,..., xic ⋅φK( ) ≡ a1,a2, ... ,aK( )
xi ≈ µ + a1φ1 + a2φ2 +...+ aKφK
a1φ1 a2φ2 aKφK...Reconstructed training face
𝑥"
![Page 52: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/52.jpg)
Face identification
Stanford University
05-Nov-2019
52
Eigenvalues (variance along eigenvectors)
![Page 53: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/53.jpg)
Face identification
Stanford University
05-Nov-2019
53
• Only selecting the top K eigenfaces à reduces the dimensionality.• Fewer eigenfaces result in more information loss, and hence less
discrimination between faces.
Reconstruction and Errors
K = 4
K = 200
K = 400
![Page 54: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/54.jpg)
Face identification
Stanford University
05-Nov-2019
54
Eigenface algorithm
• Testing1. Take query image t2. Project into eigenface space and compute projection
3. Compare projection w with all N training projections• Simple comparison metric: Euclidean• Simple decision: K-Nearest Neighbor
(note: this “K” refers to the k-NN algorithm, is different from the previous K’s referring to the # of principal components)
t→ (t −µ) ⋅φ1, (t −µ) ⋅φ2,..., (t −µ) ⋅φK( ) ≡ w1,w2,...,wK( )
![Page 55: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/55.jpg)
Face identification
Stanford University
05-Nov-2019
56
Shortcomings
• Requires carefully controlled data:– All faces centered in frame– Same size– Some sensitivity to angle
• Alternative:– “Learn” one set of PCA vectors for each angle– Use the one with lowest error
• Method is completely knowledge free– (sometimes this is good!)– Doesn’t know that faces are wrapped around 3D objects (heads)– Makes no effort to preserve class distinctions
![Page 56: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/56.jpg)
Face identification
Stanford University
05-Nov-2019
57
Summary for Eigenface
Pros• Non-iterative, globally optimal solution
Limitations
• PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination… Is there a better dimensionality reduction?
![Page 57: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/57.jpg)
Face identification
Stanford University
05-Nov-2019
58
Besides face recognitions, we can also doFacial expression recognition
![Page 58: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/58.jpg)
Face identification
Stanford University
05-Nov-2019
59
Happiness subspace (method A)
![Page 59: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/59.jpg)
Face identification
Stanford University
05-Nov-2019
60
Disgust subspace (method A)
![Page 60: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/60.jpg)
Face identification
Stanford University
05-Nov-2019
61
Facial Expression Recognition Movies (method A)
![Page 61: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/61.jpg)
Face identification
Stanford University
05-Nov-2019
62
What we will learn today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.
![Page 62: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/62.jpg)
Face identification
Stanford University
05-Nov-2019
63
Which direction will is the first principle component?
![Page 63: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/63.jpg)
Face identification
Stanford University
05-Nov-2019
64
Fischer’s Linear Discriminant Analysis
• Goal: find the best separation between two classes
Slide inspired by N. Vasconcelos
![Page 64: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/64.jpg)
Face identification
Stanford University
05-Nov-2019
65
Difference between PCA and LDA
• PCA preserves maximum variance
• LDA preserves discrimination– Find projection that maximizes scatter between classes and minimizes scatter within
classes
![Page 65: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/65.jpg)
Face identification
Stanford University
05-Nov-2019
66
Illustration of the Projection
Poor Projection
x1
x2
x1
x2
• Using two classes as example:
Good
![Page 66: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/66.jpg)
Face identification
Stanford University
05-Nov-2019
67
Basic intuition: PCA vs. LDA
![Page 67: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/67.jpg)
Face identification
Stanford University
05-Nov-2019
68
LDA with 2 variables• We want to learn a projection W such that the projection converts all the
points from x to a new space (For this example, assume m == 1):
• Let the per class means be:
• And the per class covariance matrices be:
• We want a projection that maximizes:
𝑧 = 𝑤&𝑥
![Page 68: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/68.jpg)
Face identification
Stanford University
05-Nov-2019
69
Fischer’s Linear Discriminant Analysis
Slide inspired by N. Vasconcelos
Between class scatter
Within class scatter
![Page 69: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/69.jpg)
Face identification
Stanford University
05-Nov-2019
70
LDA with 2 variablesThe following objective function:
Can be written as
![Page 70: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/70.jpg)
Face identification
Stanford University
05-Nov-2019
71
LDA with 2 variables
• We can write the between class scatter as:
• Also, the within class scatter becomes:
Slide inspired by N. Vasconcelos
![Page 71: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/71.jpg)
Face identification
Stanford University
05-Nov-2019
72
LDA with 2 variables
• We can plug in these scatter values to our objective function:
• And our objective becomes:
Slide inspired by N. Vasconcelos
![Page 72: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/72.jpg)
Face identification
Stanford University
05-Nov-2019
73
LDA with 2 variables
• The scatter variables
Slide inspired by N. Vasconcelos
![Page 73: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/73.jpg)
Face identification
Stanford University
05-Nov-2019
74
2S
1S
BS
21 SSSW +=
x1
x2Within class scatter
Between class scatter
Visualization
![Page 74: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/74.jpg)
Face identification
Stanford University
05-Nov-2019
75
Linear Discriminant Analysis (LDA)
• Maximizing the ratio
• Is equivalent to maximizing the numerator while keeping the denominator constant, i.e.
• And can be accomplished using Lagrange multipliers, where we define the Lagrangian as
• And maximize with respect to both w and λ
Slide inspired by N. Vasconcelos
![Page 75: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/75.jpg)
Face identification
Stanford University
05-Nov-2019
76
Linear Discriminant Analysis (LDA)• Setting the gradient of
With respect to w to zeros we get
or
• This is a generalized eigenvalue problem
• The solution is easy when exists
Slide inspired by N. Vasconcelos
![Page 76: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/76.jpg)
Face identification
Stanford University
05-Nov-2019
77
Linear Discriminant Analysis (LDA)
• In this case
• And using the definition of SB
• Assuming that (μ1-μ0)Tw=α is a scalar, this can be written as
• and since we don’t care about the magnitude of w
Slide inspired by N. Vasconcelos
![Page 77: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/77.jpg)
Face identification
Stanford University
05-Nov-2019
78
LDA with N variables and C classes
![Page 78: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/78.jpg)
Face identification
Stanford University
05-Nov-2019
79
Variables• N Sample images:
• C classes:
• Average of each class:
• Average of all data:
{ }Nxx ,,1 !
å==
N
kkxN 1
1µ
å=Î ikx
ki
i xN c
µ 1
![Page 79: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/79.jpg)
Face identification
Stanford University
05-Nov-2019
80
Scatter Matrices
• Scatter of class i:
å=
=c
iiW SS
1• Within class scatter:
• Between class scatter:
( )( )Tikx
iki xxSik
µµc
--= åÎ
𝑆( =)"*+
,
)-."
(𝜇" − 𝜇-)(𝜇" − 𝜇-)&
![Page 80: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/80.jpg)
Face identification
Stanford University
05-Nov-2019
81
Mathematical Formulation• Recall that we want to learn a projection W
such that the projection converts all the points from x to a new space z:
• After projection:– Between class scatter– Within class scatter
• So, the objective becomes:
WSWS BT
B =~
WSWS WT
W =~
WSW
WSW
SS
WW
TB
T
W
Bopt WW
max arg~~
max arg ==
𝑧 = 𝑤&𝑥
![Page 81: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/81.jpg)
Face identification
Stanford University
05-Nov-2019
82
Mathematical Formulation
• Solve generalized eigenvector problem:
WSW
WSWW
WT
BT
opt Wmax arg=
miwSwS iWiiB ,,1 !== l
![Page 82: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/82.jpg)
Face identification
Stanford University
05-Nov-2019
83
Mathematical Formulation
• Solution: Generalized Eigenvectors
• Rank of Wopt is limited– Rank(SB) <= |C|-1– Rank(SW) <= N-C
miwSwS iWiiB ,,1 !== l
![Page 83: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/83.jpg)
Face identification
Stanford University
05-Nov-2019
84
PCA vs. LDA
• Eigenfaces exploit the max scatter of the training images in face space
• Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter.
![Page 84: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/84.jpg)
Face identification
Stanford University
05-Nov-2019
85
Basic intuition: PCA vs. LDA
![Page 85: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/85.jpg)
Face identification
Stanford University
05-Nov-2019
86
Results: Eigenface vs. Fisherface
• Variation in Facial Expression, Eyewear, and Lighting
• Input: 160 images of 16 people• Train: 159 images• Test: 1 image
With glasses
Without glasses
3 Lighting conditions
5 expressions
![Page 86: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/86.jpg)
Face identification
Stanford University
05-Nov-2019
87
Eigenface vs. Fisherface
![Page 87: Lecture: Face Recognition](https://reader031.fdocuments.us/reader031/viewer/2022012409/616a455d11a7b741a350ade0/html5/thumbnails/87.jpg)
Face identification
Stanford University
05-Nov-2019
88
What we have learned today
• Introduction to face recognition• The Eigenfaces Algorithm• Linear Discriminant Analysis (LDA)
Turk and Pentland, Eigenfaces for Recognition, Journal of Cognitive Neuroscience 3 (1): 71–86.
P. Belhumeur, J. Hespanha, and D. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence 19 (7): 711. 1997.