SIGGRAPH Course 30:Performance-Driven Facial AnimationSIGGRAPH Course 30:Performance-Driven Facial Animation

Section:

Marker-less Face Capture and Automatic Model Construction

Part 1: Chris Bregler, NYU

Part 2: Li Zhang, Columbia University

Face Tracking ApproachesFace Tracking Approaches

• Marker-based hardware motion capture systems

• Tom Tolles (House of Moves) presentation 9:00 (earlier)

• Parag Havaldar (Sony Pictures Imageworks) presentation at 2:15 pm

Marker-based Face Capture:Marker-based Face Capture:

Marker-less Face Capture:Marker-less Face Capture:

Early Computer Face CaptureEarly Computer Face Capture

Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

• Single Camera Input

• 2D Output

• Off-line

• Interactive-Refinement

• Make-up

• Contour / Local Features

• Hand Crafted

• Linear Models / Tracking

• Disney:

Early “Markerless Facecapture”Early “Markerless Facecapture”

• Disney:

Step-Mother Eleanor Audley

Early Computer Face CaptureEarly Computer Face Capture

Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

• Single Camera Input

• 2D Output

• Off-line

• Interactive-Refinement

• Make-up

• Contour / Local Features

• Hand Crafted

• Linear Models / Tracking

Markerless Face Capture - Overview -Markerless Face Capture - Overview -

• Single / Multi Camera Input

• 2D / 3D Output

• Off-line / Real-time

• Interactive-Refinement / Face Dependent / Independent

• Make-up / Natural

• Flow / Contour / Texture / Local / Global Features

• Hand Crafted / Data Driven

• Linear / Nonlinear Models / Tracking

Common FrameworkCommon Framework

Error = Feature Error + Model Error

Tracking = Error Minimization

Difference:Difference:

Error = Feature Error + Model Error

Tracking = Error Minimization

Difference:Difference:

Error = Feature Error + Model Error

Tracking = Error Minimization

Difference:Difference:

Error = Feature Error + Model Error

Tracking = Error Minimization

Tracking = Error MinimizationTracking = Error Minimization

Kass, M., Witkin, A., & Terzopoulos, D. (1987) Snakes: Active contour models.

Tracking = Error MinimizationTracking = Error Minimization

Error = Feature Error + Model Error

Tracking = Error MinimizationTracking = Error Minimization

Error = Optical Flow + Model Error

Most general feature:

Tracking = Error MinimizationTracking = Error Minimization

Err(u,v) = || I(x,y) – J(x+u, y+v) ||

-

Basics in Optical Flow:Lucas-Kanade 1D Image

Intensity

x

u ?F G

∑ −≈x

tx xFuxF 2))()((

∑ −+=x

xGuxFuE 2))()(()(

Linearization:

Spatial Gradient Temporal Gradient

∑∈

−++=ROIyx

xGvyuxFvuE,

2))(),((),(

∑∈

−+≈ROIyx

tyx yxFvyxFuyxF,

2)),(),(),((

Spatial Gradient Temporal Gradient

ROI

ROI

(u,v)

F G

Lucas-Kanade: 2D Image

Minimize E(u,v):

0

0

=∂∂

=∂∂

vEuE

⎥⎥⎦

⎤

⎢⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡

⎥⎥⎦

⎤

⎢⎢⎣

⎡

∑∑

∑∑∑∑

yt

xt

yyx

yxx

FF

FF

v

u

FFF

FFF2

2

=>

C

D

=⎥⎦

⎤⎢⎣

⎡v

u

=⎥⎦

⎤⎢⎣

⎡v

uC

-1

D

Lucas-Kanade: Error Minimization: 2D Image

Marker-less Face Capture:Marker-less Face Capture:

In general: ambiguous using local features

-

= E(V)

Optical Flow

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

2

V

-

= E(V)

Optical Flow

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

2

V

-

= E(V)

V

Model

Optical Flow + Model

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

2

V

-

= E(V)

V

Model

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

2

V = M( )

Optical Flow + Model

V

-

V

Model

Optical Flow + linearized Model

V = M 2

Z + H V

2

Z + C

Optical Flow + 3D Model

DeCarlo, Metaxas, 1999 Eisert et al 2003

Optical Flow + MPEG4 Model

--> MediaPlayer (Eisert et al)

High-End Production:

Optical Flow + 3D Model

Disney Gemeni-ProjectWilliams et al 2002

EA Universal CaptureBorshukov et al 2002-2006

More “forgiving” Error Norm

- Faces change appearance

L2

D

More “forgiving” Error Norm

- L2 Norm vs Robust Norm

L2 robust

D D

-

Robust Error with EM layers

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

2

-

Robust Error with EM layers

I (1) - I(1) v t 1

I (2) - I(2) v t 2

I (n) - I(n) vt n

...

20.1

0.2

0.9

-

Lucas-Kanade+ changing Appearance

F G

∑ −+=x

xGuxFuE 2))()(()(

Learned PCA:

Optical Flow and PCAOptical Flow and PCA

Eigen Tracking (Black and Jepson)

2D texture and contours + PCA2D texture and contours + PCA

Active Appearance Models (AAM): (Cootes et al)

2D texture and mesh + PCA2D texture and mesh + PCA

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Lucas-Kanade + Apearance ModelsLucas-Kanade + Apearance Models

Lucas-Kanade AAMs: (Baker & Matthews)

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Affine Flow + PCA + Robust Norm Affine Flow + PCA + Robust Norm

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Disney: Gemeni-Project

Solution based on Factorization Solution based on Factorization

- We want 3 things:- 3D non-rigid shape model- for each frame:

- 3D Pose- non-rigid configuration (deformation)

-> Tomasi-Kanade-92:

W = P S

Rank 3

Solution based on Factorization Solution based on Factorization

- We want 3 things:- 3D non-rigid shape model- for each frame:

- 3D Pose- non-rigid configuration (deformation)

-> PCA-based representations:

W = P non-rigid S

Rank K

Space-Time FactorizationSpace-Time Factorization

Complete 2D Tracks or Flow Matrix-Rank <= 3*K

Nonrigid flow or Markerset -> “Rigid Stabilization + Blendshapes”

Space-Time FactorizationSpace-Time Factorization

Irani, 1999

Bregler, Hertzmann, Biermann, 2000

Torresani, Yang, Alexander, Bregler, 2001

Brand, 2001

Xiao, Kanade, 2004

Torresani, Hertzmann, 2004

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al 01,02)(Torresani et al 01,02)

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Trajectory ConstraintsTrajectory Constraints

t=2t=1

t=F

. .. .

=.

.

..

3D positions of point i for the K modes of deformation

frames

Q’ miwi : full trajectory

Space-Time Tracking Space-Time Tracking (Torresani Bregler 2002)(Torresani Bregler 2002)

QuickTime™ and aCinepak decompressor

are needed to see this picture.

• Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al)

– Rank Based Tracking

– 3D Basis Shapes

– Probabilistic Tracking / Models

– Occlusion

– Dynamical Systems

• Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al)

– Rank Based Tracking

– 3D Basis Shapes

– Probabilistic Tracking / Models

– Occlusion

– Dynamical Systems

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al 01,02)(Torresani et al 01,02)

• Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al)

– Rank Based Tracking

– 3D Basis Shapes

– Probabilistic Tracking / Models

– Occlusion

– Dynamical Systems

• Non-Rigid Models (Lorenzo Torresani, Aaron Hertzmann, et al)

– Rank Based Tracking

– 3D Basis Shapes

– Probabilistic Tracking / Models

– Occlusion

– Dynamical Systems

p ( I(pj,t ) | “point pj,t is visible”) = N ( I(pj,t )| µj ; 2 )

p ( I(pj,t ) | “pixel pj,t is an outlier”) = c

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al 01,02)(Torresani et al 01,02)

zt = A * zt-1 + nt

QuickTime™ and aCinepak decompressor

are needed to see this picture.

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al 01,02)(Torresani et al 01,02)

QuickTime™ and aCinepak decompressor

are needed to see this picture.

From Pixels to 3D Blend Shapes From Pixels to 3D Blend Shapes (Torresani et al 01,02)(Torresani et al 01,02)

Disney Gemeni ProjectDisney Gemeni Project

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Markerless Face Capture - Summary -Markerless Face Capture - Summary -

• Single / Multi Camera Input

• 2D / 3D Output

• Real-time / Off-line

• Interactive-Refinement / Face Dependent / Independent

• Make-up / Natural

• Flow / Contour / Texture / Local / Global Features

• Hand Crafted / Data Driven

• Linear / Nonlinear Models / Tracking