Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to...
Transcript of Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to...
![Page 1: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/1.jpg)
Face Recognition: from
EigenFaces to DeepFace
Artem Chernodub
IMMSP NASU
AI Ukraine, Kharkiv, 12 September 2015.
ZZ Photo
![Page 2: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/2.jpg)
Face Recognition
2 / 30
![Page 3: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/3.jpg)
Standard Face Recognition Scenario
Step 1. Face Detection Step 2. Face Alignment
Step 3. Face Matching
3 / 30
![Page 4: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/4.jpg)
Face Matching (Face Recognition)
A) Training the classifier B) Matching two face images
Classifier
Features
Answer 4 / 30
![Page 5: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/5.jpg)
Recognition of Face Attributes
• man;
• white;
• 45-50 years;
• glasses – no;
• moustache – yes;
• beard – yes.
5 / 30
![Page 6: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/6.jpg)
Beauty Recognition
http://www.linkface.cn/face/beauty Чернодуб А.Н., Пащенко Ю.А., Головченко К.А. Нейросетевая система определения привлекательности лица человека // «Нейроинформатика-2013», Москва, 21-25 января 2013, c. 254 — 259.
6 / 30
![Page 7: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/7.jpg)
Face Matching
![Page 8: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/8.jpg)
EigenFaces, Principal Component Analysis (PCA) for Face Matching, 1991
M. Turk, A. Pentland. Face Recognition using EigenFaces // Journal of cognitive
neuroscience 3 (1), p. 71-86.
• Well-known method for dimensionality reduction.
• Building features from the training data.
• Impressive visualization of the basis vectors.
8 / 30
![Page 9: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/9.jpg)
Local Binary Pattern Histograms (LBPH), 2006
• Fast & robust features for texture descriptions applied to face
recognition.
• 𝜒2 distance for comparison of histograms.
• Matrix of importance regions (weighted 𝜒2).
T. Ahonen, A. Hadid. Face Description with Local Binary Patterns:
Application to Face Recognition // IEEE Transactions on Pattern Analysis and Machine
Intelligence, Vol. 28, No. 12, p. 2037-2041.
9 / 30
![Page 10: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/10.jpg)
Deep Learning = Learning of Representations (Features)
The traditional model of pattern recognition (since the late 50's):
fixed/engineered features + trainable classifier
Hand-crafted
Feature
Extractor
Trainable
Classifier
Trainable
Feature
Extractor
Trainable
Classifier
End-to-end learning / Feature learning / Deep learning:
trainable features + trainable classifier
10 / 30
![Page 11: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/11.jpg)
Deep Feedforward Neural Networks
• 2-stage training process: i) unsupervised pre-training; ii) fine tuning
(vanishing gradients problem is beaten!).
• Number of hidden layers > 1 (usually 6-9).
• 100K – 100M free parameters.
• No (or less) feature preprocessing stage.
11 / 30
![Page 12: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/12.jpg)
FLOPS comparison
https://ru.wikipedia.org/wiki/FLOPS
Type Name Flops Cost
Mobile Raspberry Pi 1st Gen, 700
Mhz 0,04 Gflops $35
Mobile Apple A8 1,4 Gflops $700 (in iPhone 6)
CPU Intel Core i7-4930K (Ivy
Bridge), 3.7 GHz 140 Gflops $700
CPU Intel Core i7-5960X
(Haswell), 3.0 GHz 350 Gflops $1300
GPU NVidia GTX 980 4612 Gflops (single
precision), 144 Gflops
(double precision)
$600 + cost of PC
(~$1000)
GPU NVidia Tesla K80 8740 Gflops (single
precision), 2910
Gflops (double
precision)
$4500 + cost of
PC (~1500)
12 / 30
![Page 13: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/13.jpg)
Deep Face (Facebook), 2014
Y. Taigman, M. Yang, M.A. Ranzato, L. Wolf. DeepFace: Closing the Gap to Human-
Level Performance in Face Verification // CVPR 2014.
Model # of parameters Accuracy, %, LFW
Deep Face Net 128M 97.25
Human level N/A ~ 97.64
Training data: 4M facial images
13 / 30
![Page 14: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/14.jpg)
Convolutional Neural Networks: Return of Jedi
Andrej Karpathy and Fei-Fei. CS231n: Convolutional Neural Networks for Visual
Recognition http://cs231n.github.io/convolutional-networks
Yoshua Bengio, Ian Goodfellow and Aaron Courville. Deep Learning // An MIT Press
book in preparation http://www-labs.iro.umontreal.ca/~bengioy/DLbook
14 / 30
![Page 15: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/15.jpg)
DeepFace Training Framework
Step 1. Supervised training for identification
Step 2. Use produced features for face matching
Big Face database
~1-10M images,
~ 1-5K persons
Convolutional
Neural
Network
F
e
a
tu
r
e
1-5K
labels
Feature 2
Feature 1 - Cosine metric
- Euclid metric
- 𝜒2 metric
- Siamese networks
Similarity
15 / 30
![Page 16: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/16.jpg)
Databases for Deep Face Training
• Facebook’s Social Face Classification (SCF) dataset, 2014. 4.4M
labeled faces, 4030 people with 800 to 1200 faces each.
Private dataset.
• Visual Geometry Group Dataset, Oxford, 2015. 2622 people
with 1000 faces each. Private dataset.
• CASIA WebFace Database, 2014. 10575 people, 500K faces.
Public dataset.
http://www.cbsr.ia.ac.cn/english/CASIA-WebFace-Database.html
16 / 30
![Page 17: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/17.jpg)
AT&T Database (Olivetti, ORL): 1992-1994
F. Samaria, A. Harter. Parameterisation of a Stochastic Model for Human Face
Identification // Proceedings of 2nd IEEE Workshop on Applications of Computer
Vision, Sarasota FL, December 1994.
http://www.uk.research.att.com/
• Grayscale images, 92x112.
• 40 subjects, 10 images per
subject.
• Different facial expressions,
poses, glasses.
• Studio, single session.
17 / 30
![Page 18: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/18.jpg)
Face Databases
(benchmarks)
![Page 19: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/19.jpg)
FERET dataset, 1994-1996
P.J. Phillips, H. Moon, P. Rauss, S. A. Rizvi. The FERET September 1996 Database and
Evaluation Procedure // First International Conference, AVBPA'97 Crans-Montana,
Switzerland, March 12–14, 1997, pp. 395-402
http://www.nist.gov/srd/
• 2413 still facial images,
representing 856
individuals, studio, two
sessions.
• Different facial
expressions, poses, light.
19 / 30
![Page 20: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/20.jpg)
Labeled Faces in the Wild dataset, 2007
G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller. Labeled Faces in the Wild: A
Database for Studying Face Recognition in Unconstrained Environments // University
of Massachusetts, Amherst, Technical Report 07-49, October, 2007.
http://vis-www.cs.umass.edu/lfw/
• Images with faces
collected from the web.
• Pairs comparison,
restricted mode.
• test: 10-fold cross-
validation, 6000 face
pairs.
20 / 30
![Page 21: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/21.jpg)
Results for Face Matching, LWF (option “Labeled outside data”)
http://vis-www.cs.umass.edu/lfw/results.html
Method Accuracy
1 EigenFaces 60,20%
2 LBP-classic (3K features) 72,43%
3 High-dim LBP (100K features) 95,17%
4 DeepFace 97,25%
5 Human level 97,64%
6 DeepID3 99,53%
7 Baidu 99,77%
21 / 30
![Page 22: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/22.jpg)
Face Detection
![Page 23: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/23.jpg)
Viola-Jones Object Detector
• Very popular for Face Detection.
• Different features except HAAR: LBP, SURF/SIFT, etc.
• Available free in OpenCV library (http://opencv.org).
• OpenCV implementation supports Windows, Linux, Mac
OS, iOS (not officially) and Android.
• May be trained with MATLAB tool traincascade.
23 / 30
![Page 24: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/24.jpg)
Images pyramid for Viola-Jones
24 / 30
![Page 25: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/25.jpg)
FDDB: Face Detection Data Set and Benchmark
Vidit Jain and Erik Learned-Miller. FDDB: A Benchmark for Face Detection in
Unconstrained Settings // Technical Report UM-CS-2010-009, Dept. of Computer
Science, University of Massachusetts, Amherst. 2010.
http://vis-www.cs.umass.edu/fddb
• 5171 faces in a set of
2845 LWF images.
• Special software tool for
testing.
25 / 30
![Page 26: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/26.jpg)
Viola-Jones Object Detector Classifier Structure
P. Viola, M. Jones. Rapid object detection using a boosted cascade of simple features
// Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, CVPR 2001. 26 / 30
![Page 27: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/27.jpg)
FDDB: state-of-the-art
http://vis-www.cs.umass.edu/fddb/results.html
27 / 30
![Page 28: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/28.jpg)
Face Alignment
![Page 29: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/29.jpg)
3D Face Alignment
Y. Taigman, M. Yang, M.A. Ranzato, L. Wolf. DeepFace: Closing the Gap to Human-
Level Performance in Face Verification // CVPR 2014.
29 / 30
![Page 30: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/30.jpg)
2D Face Alignment
D. Yi, Z. Lei, S. Liao, S.Z. Li. Learning Face Representation from Scratch
// CoRR abs/1411.7923 (2014). [CASIA]
30 / 30
![Page 31: Face Recognition: from EigenFaces to DeepFace - AI … · Face Recognition: from EigenFaces to DeepFace ... A. Pentland. Face Recognition using EigenFaces // Journal of cognitive](https://reader031.fdocuments.us/reader031/viewer/2022020108/5b08634d7f8b9a5f6d8c5df1/html5/thumbnails/31.jpg)
Some Computer Vision / Machine Learning tools
• OpenCV, VLFeat – popular Computer Vision libraries.
• mexopencv – MATLAB interface for OpenCV. • pythonXY – all-in-one Machine Learning Package. • libSVM – reliable cross-platform Support Vector
Machine library.
• NetLab – Shallow Neural Networks & other ML models,
from Christopher Bishop’s book.
• Caffe, Theano, Torch, matconvnet – frameworks for
training Deep Neural Networks.