Mmu faculty smposium_edmond_face_modeling_v5

33
Facial Expressions - Modeling and Animation Edmond Prakash Department of Computing and Mathematics Manchester Metropolitan University United Kingdom

description

facial animation

Transcript of Mmu faculty smposium_edmond_face_modeling_v5

Page 1: Mmu faculty smposium_edmond_face_modeling_v5

Facial Expressions - Modeling and Animation

Edmond Prakash

Department of Computing and MathematicsManchester Metropolitan University

United Kingdom

Page 2: Mmu faculty smposium_edmond_face_modeling_v5

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Page 3: Mmu faculty smposium_edmond_face_modeling_v5

IntroductionIntroduction

Virtual characters in film and videoModel-based low-bandwidth teleconferencing Advanced human-computer interactionFace and facial expression recognition

Virtual surgery planning

• Recent interest in facial modeling and animation is spurred by:

• The communicative power of the face makes the realistic modeling of facial expressions an important problem in computer animation and games.

Page 4: Mmu faculty smposium_edmond_face_modeling_v5

Face Modeling for Movies

Toy Story Final Fantasy

Page 5: Mmu faculty smposium_edmond_face_modeling_v5

Face Modeling in Games

Tomb RaiderReal Human Face

Page 6: Mmu faculty smposium_edmond_face_modeling_v5

• Geometrically-based techniques– Key-frame interpolation animation

Parke (1972), Kouadio et al. (1998)– Parameterized models

Parke (1982), Thalmann et al. (1989), DiPaola (1991) – Free form deformation technique Kalra et. al. (1992), Tao and Huang (1998)

• Physically-based models– Models based on the particle system Platt (1981), Lee et al. (1995), Zhang et al. (2001)

– Finite element method Keeve et al. (1996), Koch et al. (1998)

Previous WorkPrevious Work(Facial animation)

Page 7: Mmu faculty smposium_edmond_face_modeling_v5

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Page 8: Mmu faculty smposium_edmond_face_modeling_v5

Facial Expression Recognition

• Prof. Eric Sung at NTU also came up with a request like this:

• Given several hours of video, how will you efficiently recognize the face of former US president Bill Clinton with a smiling expression?

• We developed a 3D Model based approach.• PhD Thesis of Zhang Yu

Page 9: Mmu faculty smposium_edmond_face_modeling_v5

Individualized Face Modeling and AnimationIndividualized Face Modeling and Animation

Reconstructed Facial SurfaceReconstructed Facial Surface

Multi-layer MSD Deformable Skin Model

Multi-layer MSD Deformable Skin Model

Generic Skull Model

Physical Facial Muscle ModelsPhysical Facial Muscle Models

Muscle Construction

Muscle Construction

Anatomy-based Personalized Face Model

Anatomy-based Personalized Face Model

Facial Action Editor

Facial Action Editor

ODE SolverODE

Solver

Texture Mapping and

Rendering

Texture Mapping and

Rendering

Synthesized ExpressionsSynthesized Expressions

Skull FittingSkull

Fitting

Facial Data Acquisition & Preprocessing

Facial Data Acquisition & Preprocessing

Range Data

Range Data

Reflectance ImagesReflectance Images View-based Texture

Extraction

View-based Texture

Extraction

Page 10: Mmu faculty smposium_edmond_face_modeling_v5

Simulation ResultsSimulation Results

Happiness Anger

How do we mathematically define this?

Page 11: Mmu faculty smposium_edmond_face_modeling_v5

Simulation ResultsSimulation Results

Surprise Sadness Disgust

Page 12: Mmu faculty smposium_edmond_face_modeling_v5

Expression Simulation on Real Person

Page 13: Mmu faculty smposium_edmond_face_modeling_v5

Simulation ResultsSimulation Results

System Performance

Computer program can generate happiness easily ie 34.8 frames/sec.Sadness is difficult to synthesize. Needs more muscles and more time.

SO BE HAPPY ALWAYS!

Page 14: Mmu faculty smposium_edmond_face_modeling_v5

• Examples of the simulated expressions on various personalized facial models.

• Real-time: Physics based Skin, Muscle, Skull

• Examples of the simulated expressions on various personalized facial models.

• Real-time: Physics based Skin, Muscle, Skull

Simulation ResultsSimulation Results

Page 15: Mmu faculty smposium_edmond_face_modeling_v5

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Page 16: Mmu faculty smposium_edmond_face_modeling_v5

Face model adaptation

• Scanned face not a good mesh.

• Adapt from a template mesh.

• Train the RBF with the corresponding feature points

Template head

Scanned head

Page 17: Mmu faculty smposium_edmond_face_modeling_v5

RBF and template face adaptation

• What is RBF?

Sourcepoint

Target point

Weight coefficient

Basic function

Rotation, shearing, scaling matrix

Translation vector

Page 18: Mmu faculty smposium_edmond_face_modeling_v5

Template model adaptation

• Train the RBF with the corresponding feature points

• Solve the linear equation AX=B by LU decomposition

• Apply all the vertex in the template model with the solved weights and R, t to the RBF

Page 19: Mmu faculty smposium_edmond_face_modeling_v5

Radial basis function result

Adapted head

Page 20: Mmu faculty smposium_edmond_face_modeling_v5

Overview

• Introduction• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions• Summary

Page 21: Mmu faculty smposium_edmond_face_modeling_v5

Subtle facial expressions

• Summer of 2003 • Beckman Institute at UIUC• Prof. Yoshisa Shinagawa • Study the subtle facial expressions.

• Subtle Expressions – Experiments by Ramya Sridharan

Page 22: Mmu faculty smposium_edmond_face_modeling_v5

How to Quantify Subtle Expressions?(Synthesized variations of happiness )

Laugh Devious smile

Dampened smile Duchenne simle

Page 23: Mmu faculty smposium_edmond_face_modeling_v5

Observation

frame1 frame 15 Superimposed [(a) + (b)] /2

Page 24: Mmu faculty smposium_edmond_face_modeling_v5

Dense Optical Flow along U & V directions

Uflow Vflow

Resultant

Page 25: Mmu faculty smposium_edmond_face_modeling_v5

Experiment 2: Flow analysis for inside features (half face, eyes, one eye, cheek, mouth and lips)

reduce processing time

Page 26: Mmu faculty smposium_edmond_face_modeling_v5

Optic Flow

U Matrix

V Matrix

UV Matrix: To Quantify Subtle Expressions

Page 27: Mmu faculty smposium_edmond_face_modeling_v5

+ [U][V]

Page 28: Mmu faculty smposium_edmond_face_modeling_v5

U1,V1

U2,V2

U3,V3

User text input

Input image

text input from the user and generate the corresponding output face according to the user’s input.

Page 29: Mmu faculty smposium_edmond_face_modeling_v5

Human Skin Rendering Real vs Synthetic

Page 30: Mmu faculty smposium_edmond_face_modeling_v5

Taxonomy of Face Animation

YoungFace

Speech to face animation

Speech stream Text stream

Text to face animation

Video to face animation

Face Modeling

Expressions

Expression to face animation

Video stream

Page 31: Mmu faculty smposium_edmond_face_modeling_v5

Challenges

• Our goal is to use – Face in Games

• Practical Applications– Visual Synthesis of Subtle expressions– Visual Expression Analysis– Expression Cloning (2D images)– Expression Cloning on to 3D Heads– Partial Physical Disabilities & Plastic Surgery

Movie/Game Characters

Page 32: Mmu faculty smposium_edmond_face_modeling_v5

What is happening elsewhere in 2008?

• SIGGRAPH 2008

• VR 2008

Page 33: Mmu faculty smposium_edmond_face_modeling_v5

Conclusion• Physics Based Face Animation• Face Adaptation• Subtle Facial Expressions

Face processing still needs lot of research.

If you would like to know more:

Please send me email at: [email protected] visit:

http:/www.docm.mmu.ac.uk/STAFF/E.Prakash