Deformation Deformation Modeling for Robust Modeling for Robust

3D Face Matching3D Face Matching

Xioguang Lu and Anil K. JainXioguang Lu and Anil K. JainDept. of Computer Science & EnginDept. of Computer Science & Engin

eeringeeringMichigan State UniversityMichigan State University

ProblemProblem

Although 3D facial scans do not vary wAlthough 3D facial scans do not vary with lighting or pose changes, nonrigid fith lighting or pose changes, nonrigid facial deformations can hurt recognitioacial deformations can hurt recognitionn

Collecting and storing multiple expresCollecting and storing multiple expression template scans for each subject is sion template scans for each subject is not practicalnot practical

Expressions can have differing intensitExpressions can have differing intensitiesies

Proposed SchemeProposed Scheme

A (hierarchical) geodesic sampling is A (hierarchical) geodesic sampling is used to quantify facial expressionused to quantify facial expression

Expression variations are learned from a Expression variations are learned from a small control groupsmall control group

These variations are used to create a These variations are used to create a deformable model from gallery deformable model from gallery templatestemplates

This deformable model is fit to the target This deformable model is fit to the target scan and matching distance computedscan and matching distance computed

SamplingSampling

Landmarks are manually selected Landmarks are manually selected (nose tip, eye corners, mouth (nose tip, eye corners, mouth corners, and mouth contour)corners, and mouth contour)

Geodesic distance between certain Geodesic distance between certain features is computed (hierarchically features is computed (hierarchically in latest work)in latest work)

Geodesics are split into L segments Geodesics are split into L segments of equal length to generate L-1 new of equal length to generate L-1 new feature pointsfeature points

Deformation TransferDeformation Transfer Register non-neutral scan with neutral scan of sRegister non-neutral scan with neutral scan of s

ame face to estimate landmark displacementame face to estimate landmark displacement Establish a mapping Establish a mapping ΦΦ from the neutral gallery from the neutral gallery

to the neutral target faceto the neutral target face Use Use ΦΦ to transfer landmarks in the non-neutral to transfer landmarks in the non-neutral

gallery scan to the (synthesized) non-neutral targallery scan to the (synthesized) non-neutral targetget

Establish a mapping Establish a mapping ψψ from the neutral to non- from the neutral to non-neutral targetneutral target

Interpolate Interpolate ψψ using thin-plate-spline mapping using thin-plate-spline mapping Boundary constraints are included in thin-platBoundary constraints are included in thin-plat

e-spline calculation as additional landmark poie-spline calculation as additional landmark pointsnts

RegistrationRegistration

Neutral and non-neutral target are aligNeutral and non-neutral target are aligned using features which don’t move ned using features which don’t move much with expression changes, such amuch with expression changes, such as eye corners and nose tips eye corners and nose tip

This separates rigid transformations frThis separates rigid transformations from nonrigid transformationsom nonrigid transformations

Thin-Plate SplinesThin-Plate Splines Goal: find a mapping from landmark seGoal: find a mapping from landmark se

t U to V with known correspondencest U to V with known correspondences Method: imagine V as a thin metal sheeMethod: imagine V as a thin metal shee

t and find a function which minimizes bt and find a function which minimizes bending energyending energy

Solution: F(u) = c + A*u + WSolution: F(u) = c + A*u + WTT*s(u)*s(u) s(u) = (|u – us(u) = (|u – u11|, |u – u|, |u – u22|, …)|, …)TT

An analytical solution can be obtained for 3An analytical solution can be obtained for 3D pointsD points

Deformable Model Deformable Model ConstructionConstruction

To generate a deformable model, each learned To generate a deformable model, each learned expression is simulated on a neutral gallery faceexpression is simulated on a neutral gallery face

Face is represented as a combination of shape vFace is represented as a combination of shape vectors:ectors:

M is the number of synthesized templates, M is the number of synthesized templates, αα ii is the w is the w

eight of each templateeight of each template By adjusting the weights By adjusting the weights αα ii, various combinatio, various combinatio

ns of expressions can be generatedns of expressions can be generated To reduce computational complexity, one deforTo reduce computational complexity, one defor

mable model per expression is generatedmable model per expression is generated

MatchingMatching Coarse alignment performed as during deformaCoarse alignment performed as during deforma

tion transfertion transfer Alignment refined with iterative closest point alAlignment refined with iterative closest point al

gorithmgorithm Associate each point with nearest neighbor, calculate Associate each point with nearest neighbor, calculate

transform to minimize distance, repeattransform to minimize distance, repeat Minimize a cost function by solving for Minimize a cost function by solving for αα iiss

R and T are rotation and translation matrices, S is the R and T are rotation and translation matrices, S is the

deformable model, and Sdeformable model, and Stt is the test scan is the test scan Use these Use these αα iis to compute a new iterative closest s to compute a new iterative closest

point distance, and return to step 2 until converpoint distance, and return to step 2 until convergencegence

Experiment IExperiment I

Self-collected database of 10 subjects at Self-collected database of 10 subjects at 3 different poses, with 7 different 3 different poses, with 7 different expressions, for 210 total scans and 10 expressions, for 210 total scans and 10 gallery modelsgallery models

5 subjects at random chosen as control 5 subjects at random chosen as control group, leaving 105 scans for recognitiongroup, leaving 105 scans for recognition

Results:Results:

Experiment IIExperiment II

Control group: 10 subjects from Control group: 10 subjects from Experiment IExperiment I

Test group: 90 additional subjects, Test group: 90 additional subjects, with 6 scans each at different with 6 scans each at different viewpoints (in most cases)viewpoints (in most cases) 533 total test scans533 total test scans

Results:Results:

Experiment IIIExperiment III A subset of FRGC v2.0 datasetA subset of FRGC v2.0 dataset Scans with the earliest timestamp and neutral Scans with the earliest timestamp and neutral

expression are used as templatesexpression are used as templates 50 gallery scans, 150 test scans50 gallery scans, 150 test scans 10 subjects in Experiment I used as control group10 subjects in Experiment I used as control group Latest results (after publication): Latest results (after publication):

ConclusionsConclusions

One area for improvement (noted in One area for improvement (noted in the paper) was the dependence on the paper) was the dependence on manual landmark labelingmanual landmark labeling

Also, I thought that there might be Also, I thought that there might be some application of geometric some application of geometric invariants to replace their invariants to replace their registration step (which is subject to registration step (which is subject to local minima)local minima)

Questions?Questions?