Sami Romdhani Volker Blanz Thomas Vetter University of Freiburg Supported by DARPA
description
Transcript of Sami Romdhani Volker Blanz Thomas Vetter University of Freiburg Supported by DARPA
Face Face IdentificationIdentificationby by FittingFitting a a
3D3D Morphable Model Morphable Modelusing using LinearLinear Shape and Texture Error Shape and Texture Error
FunctionsFunctions
Sami Romdhani Volker Blanz Thomas Vetter
University of Freiburg
Supported by DARPA
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 2/26
The ProblemThe Problem
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 3/26
MenuMenu
Historical Methods
3D Morphable Model
LiST : a Novel Fitting Algorithm
Identification Experiments on more than 5000 Images
Identification Confidence = Fitting Accuracy
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 4/26
Historical Methods : Historical Methods : Active Appearance ModelActive Appearance Model
Use of a generative model:
1. View based (2D), Correspondence basedex: AAM of Cootes and Taylor
Drawbacks:- small pose variation statistically
modeled !
- large pose var. necessitates many models !
- illumination not addressed !
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 5/26
Historical Methods : Illumination ConeHistorical Methods : Illumination Cone
2. Shape from Shading= Recovering 3D shape from Illumination variations
ex: Illumination Cone of Georghiades, Belhumeur & Kriegman
Limited use : up to 24° azimuth variation !
Drawback:Impractical: requires many imagesRestrictive assumptions : constant
albedo, lambertian,no cast shadows
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 6/26
3D Shape
3D Morphable Model - Key Features 13D Morphable Model - Key Features 1
1. Representation = 3D Shape + Texture Map
Texture Map
s
t
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 7/26
3D Morphable Model - Key Features 23D Morphable Model - Key Features 2
2. Accurate & Dense Correspondence
PCA accounts for intrinsic ID parameters only
2 3 4 1s S α
4 3 2 1t T β
...
...
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 8/26
3D Morphable Model - Key Features 33D Morphable Model - Key Features 3
3. Extrinsic parameters modeled using Physical Relations:- Pose : 3x3 Rotation matrix
- Illumination : Phong shading accounts for cast shadows and specular highlights
No Lambertian Assumption.
, ,
1 0 0
0 1 0xk
kk y
tx
yf
t
αR S
( )k
ambient dir speculark k k k
k
r
g
b
A TA aβ
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 9/26
3D Morphable Model - Key Features 43D Morphable Model - Key Features 4
4. Photo-realistic images rendered using Computer Graphicsα
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 10/26
Model Fitting : DefinitionModel Fitting : Definition
IterativeModelFitting
,α β,ρ
ModelRenderin
g
( , )mI x y
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 11/26
Model Fitting - History : Standard Optimization Model Fitting - History : Standard Optimization TechniquesTechniquesJones, Poggio 98 : Gradient DescentBlanz, Vetter 99 : Stochastic Gradient DescentPighin, Szeliski, Salesin 99 : Levenberg-Marquardt
-
2
2
2
I
I
I
Model EstimateInput
Difference I
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 12/26
Model Fitting - History : Image Difference Model Fitting - History : Image Difference DecompositionDecomposition
IDD introduced by Gleicher in 97 and used by Sclaroff et al. in 98, and Cootes et al. in 98
-
I
A
Input
Difference I
Model Estimate
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 13/26
LiST : Non-linearity LiST : Non-linearity
1. Non-linear warping
2. Non-linear parametersinteraction
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 14/26
LiST : Shape & Texture Parameters recoveryLiST : Shape & Texture Parameters recovery
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
( , , , , )w x y r g b ( , )mI x y
, ,x y zn n n
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 15/26
( , , , , )w x y r g b
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
LiSTLiST
( , )mI x y
( , )I x y
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 16/26
Optical Flow
α
S
, , R
1 0 0,
0 1 0f
,xt yt
, ,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
( , )mI x y
( , )I x y
LiST : Optical FlowLiST : Optical Flow
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 17/26
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
Optical Flow( , )mI x y
( , )I x y
Lev.-Mar.
LiST : Rotation, Translation & Size RecoveryLiST : Rotation, Translation & Size Recovery
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 18/26
α
S
, , R
1 0 0,
0 1 0f
,xt yt
,,x y z
, ,x y z
,x y
β
T
, ,ambient dir specularaA A
, ,r g b
, ,r g b
, ,x y zn n n
Optical Flow( , )mI x y
( , )I x y
Lev.-Mar.Lev.-Mar.
LiST : Illumination RecoveryLiST : Illumination Recovery
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 19/26
LiST : DiscussionLiST : Discussion
• Shape and Texture recoveries are interleavedThe recovery of one helps the recovery of the other
• Takes advantage of the linear parts of the model
• Recovers out-of-the-image-plane rotation & directed illumination
• 5 times faster than Stochastic Gradient Descent
Drawbacks:• Still requires manual initialization• Still not fast enough
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 20/26
Experiments : The CMU-PIE Face DatabaseExperiments : The CMU-PIE Face Database
• Publicly available
• Systematic pose & illumination variations
• 68 Individuals
• 4488 Images with combined Pose & Illumination var.
• 884 Images with Pose var.
-20
-15
-10
-5
0 0
5
10
15
20
-5
0
5 head
flashescamerashead
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 21/26
Experiments : FittingExperiments : Fitting
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 22/26
Experiments : Identification across PoseExperiments : Identification across Pose
Identification Results across Pose
0102030405060708090
34 31 14 11 29 9 27 7 5 37 25 2 22
Gallery Pose
Pe
rce
nta
ge
LiST, average=76%FaceIt, average=21%
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 23/26
Experiments : Identification across Illumination & Experiments : Identification across Illumination & PosePose
Identification on 4488 imagesacross Pose & Illuminationaveraged over Illumination
Front Side Profile
Front 97 91 60
Side 93 96 71
Profile 65 71 86
Gallery
Probe
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 24/26
Identification Confidence : TheoryIdentification Confidence : Theory
Can we be sure to have correctly identified someone ?
Identification Confidence depends mostly on the Fitting
We think:
Classification Support Vector MachineInput:Mahalanobis distance from the average
SSE over 5 regions of the face
Output: Good Fitting Y/N ?
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 25/26
-2-1.25-0.75-0.250.250.751.2520
5
10
15
20
25
30
35
Fitting Score = SVM Output
% o
f Exp
erim
ents
29 % 33 % 12 % 6 % 4 % 7 % 7 % 3 %
Iden
tific
atio
n P
erce
ntag
e
Identification vs. Fitting Score
97.4 %95.1 %
83.7 %
76.5 %
58.9 %
43.2 %
38.2 %
26.8 %
20
30
40
50
60
70
80
90
100
Identification Confidence : ResultIdentification Confidence : Result
The model is goodwe only need to improve the fitting accuracy
7th ECCV – 31 May 2002 - Volume 4, pp 3 - 19 26/26
ConclusionsConclusions
Novel Fitting Algorithm :• Use of Optical Flow to recover a Shape Error• Recovers most of the parameters linearly• Recovers a few non-linear parameters using
Lev.-Mar.
State of the art identification performances across
Pose & Illumination
Drawbacks:• Still not fast enough• Still requires manual initialisation