Facial Expressions - Modeling and Animation

Edmond Prakash

Department of Computing and MathematicsManchester Metropolitan University

United Kingdom

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

IntroductionIntroduction

Virtual characters in film and videoModel-based low-bandwidth teleconferencing Advanced human-computer interactionFace and facial expression recognition

Virtual surgery planning

• Recent interest in facial modeling and animation is spurred by:

• The communicative power of the face makes the realistic modeling of facial expressions an important problem in computer animation and games.

Face Modeling for Movies

Toy Story Final Fantasy

Face Modeling in Games

Tomb RaiderReal Human Face

• Geometrically-based techniques– Key-frame interpolation animation

Parke (1972), Kouadio et al. (1998)– Parameterized models

Parke (1982), Thalmann et al. (1989), DiPaola (1991) – Free form deformation technique Kalra et. al. (1992), Tao and Huang (1998)

• Physically-based models– Models based on the particle system Platt (1981), Lee et al. (1995), Zhang et al. (2001)

– Finite element method Keeve et al. (1996), Koch et al. (1998)

Previous WorkPrevious Work(Facial animation)

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Facial Expression Recognition

• Prof. Eric Sung at NTU also came up with a request like this:

• Given several hours of video, how will you efficiently recognize the face of former US president Bill Clinton with a smiling expression?

• We developed a 3D Model based approach.• PhD Thesis of Zhang Yu

Individualized Face Modeling and AnimationIndividualized Face Modeling and Animation

Reconstructed Facial SurfaceReconstructed Facial Surface

Multi-layer MSD Deformable Skin Model

Multi-layer MSD Deformable Skin Model

Generic Skull Model

Physical Facial Muscle ModelsPhysical Facial Muscle Models

Muscle Construction

Muscle Construction

Anatomy-based Personalized Face Model

Anatomy-based Personalized Face Model

Facial Action Editor

Facial Action Editor

ODE SolverODE

Solver

Texture Mapping and

Rendering

Texture Mapping and

Rendering

Synthesized ExpressionsSynthesized Expressions

Skull FittingSkull

Fitting

Facial Data Acquisition & Preprocessing

Facial Data Acquisition & Preprocessing

Range Data

Range Data

Reflectance ImagesReflectance Images View-based Texture

Extraction

View-based Texture

Extraction

Simulation ResultsSimulation Results

Happiness Anger

How do we mathematically define this?

Simulation ResultsSimulation Results

Surprise Sadness Disgust

Expression Simulation on Real Person

Simulation ResultsSimulation Results

System Performance

Computer program can generate happiness easily ie 34.8 frames/sec.Sadness is difficult to synthesize. Needs more muscles and more time.

SO BE HAPPY ALWAYS!

• Examples of the simulated expressions on various personalized facial models.

• Real-time: Physics based Skin, Muscle, Skull

• Examples of the simulated expressions on various personalized facial models.

• Real-time: Physics based Skin, Muscle, Skull

Simulation ResultsSimulation Results

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Face model adaptation

• Scanned face not a good mesh.

• Adapt from a template mesh.

• Train the RBF with the corresponding feature points

Template head

Scanned head

RBF and template face adaptation

• What is RBF?

Sourcepoint

Target point

Weight coefficient

Basic function

Rotation, shearing, scaling matrix

Translation vector

Template model adaptation

• Train the RBF with the corresponding feature points

• Solve the linear equation AX=B by LU decomposition

• Apply all the vertex in the template model with the solved weights and R, t to the RBF

Radial basis function result

Adapted head

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Subtle facial expressions

• Summer of 2003 • Beckman Institute at UIUC• Prof. Yoshisa Shinagawa • Study the subtle facial expressions.

• Subtle Expressions – Experiments by Ramya Sridharan

How to Quantify Subtle Expressions?(Synthesized variations of happiness )

Laugh Devious smile

Dampened smile Duchenne simle

Observation

frame1 frame 15 Superimposed [(a) + (b)] /2

Dense Optical Flow along U & V directions

Uflow Vflow

Resultant

Experiment 2: Flow analysis for inside features (half face, eyes, one eye, cheek, mouth and lips)

reduce processing time

Optic Flow

U Matrix

V Matrix

UV Matrix: To Quantify Subtle Expressions

+ [U][V]

U1,V1

U2,V2

U3,V3

User text input

Input image

text input from the user and generate the corresponding output face according to the user’s input.

Human Skin Rendering Real vs Synthetic

Taxonomy of Face Animation

YoungFace

Speech to face animation

Speech stream Text stream

Text to face animation

Video to face animation

Face Modeling

Expressions

Expression to face animation

Video stream

Challenges

• Our goal is to use – Face in Games

• Practical Applications– Visual Synthesis of Subtle expressions– Visual Expression Analysis– Expression Cloning (2D images)– Expression Cloning on to 3D Heads– Partial Physical Disabilities & Plastic Surgery

Movie/Game Characters

What is happening elsewhere in 2008?

• SIGGRAPH 2008

• VR 2008

Conclusion• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions

Face processing still needs lot of research.

If you would like to know more:

Please send me email at: E.Prakash@mmu.ac.ukOr visit:

http:/www.docm.mmu.ac.uk/STAFF/E.Prakash