Data Insufficiency in Sketch Versus Photo Face...
Transcript of Data Insufficiency in Sketch Versus Photo Face...
![Page 1: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/1.jpg)
Data Insufficiency in Sketch Versus Photo Face Recognition
Jonghyun ChoiAbhishek Sharma, David W. Jacobs, Larry S. Davis
Ins=tute of Advanced Computer StudiesUniversity of Maryland, College Park
CVPR Workshop in Biometrics 2012
17 June 2012
![Page 2: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/2.jpg)
Sketch-Photo Face Recognition•Why is it important?-‐ Automated criminal search by forensic sketch can reduce the 8me of crime inves8ga8on
Matching
GalleryProbe
Image Courtesy by B. Klare from “Matching Forensic Sketches to Mug Shot Photos”, PAMI 2011
![Page 3: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/3.jpg)
Popular Benchmark in Literature
• CUFS dataset[1]
-‐ Public benchmark dataset-‐ Promo8ng the ini8al research‣ A controlled dataset
✓Well lit photos, neutral expression, frontal poses
-‐ Many approaches evaluated so far
[1] Tang and Wang, “Face Photo Recogni8on using Sketch”, ICIP 2002
![Page 4: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/4.jpg)
Timeline of Research
2002
First CUFS dataset released (188) Baseline
ICIP2003
CUFS dataset expanded (608),
Bayesian Approach
ICCV
Non-linear approach
CVPR
Baseline
T.CSVT
Journal Extension
2006ECCV
Common Discriminant
Feature Extraction
2007 2008 2010 2011 2012ICASSP
E-HMM and selected ensemble
T.CSVT
E-HMM and selected ensemble
Journal Extension
ECCV
Lighting and Pose Invariant
CUFS dataset expanded (1,800),
Coupled Information
Theoretic Encoding
CVPR
Coupled Spectral Regression
2009CVPR
20052004
Local Feature based Discriminant
Analysis (LFDA)
PAMI
• On the CUFS Dataset
65
76.667
88.333
100
Accuracy
![Page 5: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/5.jpg)
Summary of Previous ResultsApproach #Train #Test Rate (%)
Sketch SynthesisTang and Wang 88 100 71Tang and Wang 306 300 81.3
Nonlinear 306 300 87.67E-HMM 306 300 95.24
MS MRF+LDA 306 300 96.3MS MRF+LDA 88 100 96
MS MRF+W.PCA 88 100 99Modelling Modality Gap
PLS-subspace 88 100 93.6Klare et al. 306 300 99.47
CITP 306 300 99.87
Nearly perfect result!
Is the problem solved?
![Page 6: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/6.jpg)
Yes
![Page 7: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/7.jpg)
Yes CUFS Datasetfor
![Page 8: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/8.jpg)
CUFS Dataset• 606 photo-‐sketch pairs-‐ Random par88oning for training/tes8ng
• Combined with CUFSF (2011) from FERET-‐ Total 1,800 photo-‐sketch pairs• viewed sketch dataset-‐ Provides well-‐aligned photo-‐sketch pairs‣ Good for analysis of difference in sketch
and photo domains without any other factors’ interven8ons
• viewed sketch dataset
![Page 9: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/9.jpg)
CUFS Dataset (Cont’d)
• viewed sketch dataset-‐ Sketch is drawn by ar8sts‣ Capturing subtle edge similarity (e.g. hair style)-‐ Pre-‐processed to make them well aligned as well
![Page 10: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/10.jpg)
CUFS Dataset (Cont’d)
• viewed sketch dataset-‐ Sketch is drawn by ar8sts‣ Capturing subtle edge similarity (e.g. hair style)-‐ Pre-‐processed to make them well aligned as well
![Page 11: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/11.jpg)
Real-World Scenario for Sketch-Photo Face Recognition1. Eye-‐witness describes the criminal’s
facial traits verbally2. Forensic ar;sts draw the sketch
according to the verbal descrip;on• Forensic sketch[1]
[1] B. Klare et al.,”Matching Forensic Sketches to Mug Shot Photos”, PAMI 2011
➡ Viewed sketch is not realistic
*Images courtesy from B. Klare
![Page 12: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/12.jpg)
Insufficiency of the CUFS Dataset• Well-‐alignment of CUFS dataset-‐ Good for ini8al research on domain difference
(photo-‐sketch) w/o interven8on of other factors-‐ But simplifies the problem too much‣ Simple edge matching techniques might work
‣ Ignores true variability of sketch-‐face recogni8on:
✓ Mis-‐alignment of fiducial components
✓ Seman8c descrip8on of shapes of fiducial component
✓ No precise descrip8on for subtle difference (e.g. hair style)
![Page 13: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/13.jpg)
Today, We Show• To obtain good result on CUFS dataset-‐ Discrimina8ve edge matching technique is
enough‣ Outperforms state of the art in face
iden8fica8on sebng-‐ No effort in reducing modality gap is required‣ Thus no training set except gallery set is
required
‣ But even outperforms state of the art in bigger set
![Page 14: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/14.jpg)
Discriminative Edge Matching
• Edge features-‐ Gabor wavelet response‣ Blurred edge tendency: Macro edge
-‐ CCS-‐POP[1]
‣Micro-‐edgelet
• Discrimina8ve weight on the feature-‐ Build a one-‐vs-‐all PLS model
[1] Choi et al.,”A Complementary Local Feature for Face Iden8fica8on”, WACV 2012
![Page 15: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/15.jpg)
Partial Least Sqaures• A supervised dimension reduc8on technique by
maximizing covariance of weighted independent variable (X) and weighted dependent variable (Y)
• Using NIPALS algorithm[1] to obtain the regression solu8on from X to Y
[1] H. Wold, Par8al Least Squares, 1985
feature label
w = max
|w|=1cov(Xw, Y )
2
![Page 16: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/16.jpg)
Overall System Diagram[1]
GalleryBuild “One-‐vs-‐All” PLS regression models
Posi=ve Samples
Nega=ve Samples
…Model
A…
Posi=ve Samples
Nega=ve Samples
…
ModelZ
…
…
A B
DE
F
G
Z
Tes=ng Regression responses
Model A Model B Model C Model D Model Z
…
C
Iden=fica=on Result
…
Model Building (Training)
PLSR
PLSR
ProbeRegression
C
[1] Schwartz et al., “A Robust and Scalable Approach to Face Iden8fica8on”, ECCV 2010
![Page 17: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/17.jpg)
Experimental Setup• CUFS+CUFSF (CUFS) dataset-‐ 1,800 pairs of sketch-‐face-‐ No extra training set: Use all pairs for test-‐ Photo to Sketch / Sketch to Photo experiments
• Comparison to previous work• Various image cropping-‐ Tight/Loose crop-‐ Horizontal/Ver8cal strip crop-‐ Fiducial component crop
![Page 18: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/18.jpg)
Experimental ResultsApproach #Train #Test Rate (%)
Sketch SynthesisTang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94
![Page 19: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/19.jpg)
• Test with 100 samples
70
77.5
85
92.5
100
71
9699
93.6
100
2002 [24]2006 [26] 2010 [31]
2010 [31]2011
Ours
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our Method
Experimental Results
![Page 20: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/20.jpg)
Experimental Results• Test with 300 samples
80
85
90
95
100
81.3
87.67
95.24 96.3
99.47 99.87 100
2003 [25]2005 [13]
2007,8 [4,35]2009 [30]
2011 [9]2011 [32]
Ours (G+C)
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our Method
![Page 21: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/21.jpg)
Experimental Results• Test with 1,800 samples-‐ Compared to the results tested with 300 samples
99.4
99.55
99.7
99.85
100
99.47
99.8799.94
2011 [9] w/3002011 [32] w/300
Ours (G+C) w/1800
Accuracy (%)
Approach #Train #Test Rate (%)Sketch Synthesis
Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3
Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24
MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99
Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87
Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90
Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50
Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94
Our Method
![Page 22: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/22.jpg)
Different Cropping
• Tight/Loose Crop
• Horizontal/Ver8cal Strip Crop
• Fiducial Component Crop
![Page 23: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/23.jpg)
Results on Tight/Loose Cropping
97
97.75
98.5
99.25
100
97.6
98.8
99.94
TightMedium
Loose
Accuracy (%)
![Page 24: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/24.jpg)
Results on Fiducial Component Cropping
77
82.75
88.5
94.25
10095.11
77.67 79.585.22
Ocular Nose MouthHair
Accuracy
![Page 25: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/25.jpg)
Results on Strip Cropping
77
82.75
88.5
94.25
100
1 2 3 4
H V Hc Vc
![Page 26: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/26.jpg)
Discussion & Conclusion
• A simple discrimina8ve edge analysis can perform well in overly-‐reduced problem of sketch-‐photo matching
• Now is the 8me to move on to more challenging dataset
•We suggest a guideline for new dataset (Please refer to our paper)
![Page 27: Data Insufficiency in Sketch Versus Photo Face Recognitionjhchoi/paper/cvprw2012_sketch_slides.pdf · Data Insufficiency in Sketch Versus Photo Face Recognition JonghyunChoi AbhishekSharma,(David(W.(Jacobs,(Larry(S.(Davis](https://reader033.fdocuments.us/reader033/viewer/2022050202/5f5626a00f17bd4f6475bb89/html5/thumbnails/27.jpg)
Q/AThank you!
Authors are supported byMURI Grant N00014-08-10638 from Office of Naval Research