Use and Re-use of Facial Motion Capture
M. Sanchez, J. Edge, S. King and S. Maddock
0. Motivation Facial Animation in the Computer Graphics industry
is a mainly human-driven process, requiring a lot of time and resources
Other aspects of Character Animation, such as Skeletal Animation, have been successfully automated by the use of human motion capture technology
Applying the same approach to Facial Animation could sensibly reduce the workload involved, and would lead to a corresponding increase in the realism of Facial Animation
However, the differences in the nature of the captured content requires the development of specific techniques to make Facial Motion Capture applicable
1. System overview
1.1 Facial Motion Capture System input:
Problems: It doesn’t analyze the motion
over the full geometry of the face (just at the markers);
The captured face may not correspond with the face to be animated;
Noise and missing data.
3D tracking of a predefined set of markers attached to the skin surface
2. Animating the skin
2. Animating the skin How to reconstruct the deformation
of the complete skin surface when only the movement of a few points is known?
Direct interpolation of the movement of the markers over the skin
(Kshirsagar et al. 00, Pasquariello and Pelachaud. 01)
Dirichlet Free Form Deformations (Escher et al. 98)
Radial Basis Functions (Fidaleo, Noh et al. 00)
We use Planar Bones (Sanchez and Maddock 03)
2.1. Planar Bones Extended formulation of Surface-oriented
Free Form Deformations(Kokkevis and Singh, 00)
Define a parameterisation of every vertex over a control mesh, used to drive a deformation
Preserve a “distance relation” between the control structure and the deformed geometry
Replicate proper transmission of motion across the skin without the need of surface metrics
4. Retargeting Facial Motion Capture
The dimensions of the face are different, and so is the scale of the motion;
Conventional full-body Motion Capture retargeting is not applicable;
The correspondence between different faces is highly non-linear.
4. Building a mapping between faces
4. Building a mapping between faces
Retargeting FMC requires:
1) Adapting the Planar Bones control mesh to the target geometry;
2) Scaling the motion of the markers according to the change of physiognomy.
Ideally, both processes should be performed automatically
In practical terms, we need some user input.
4.1. Fitting the control mesh
4.1. Fitting the control mesh 3 stages:
1) Radial Basis Functions -
produce initial approximation
2) Cylindrical projection –
Constraining the markers to the target surface
3) Mesh fitting – Blind constrained optimisation
Additional parameters of the Planar Bones method are also retargeted by this process:
Extents of the deformation (affection volume) Discontinuity maps
4.1. Fitting the control mesh RBF stage:
Build an interpolant of the offset between the markers labelled on the target face and their equivalents in the reference model
Evaluate this function at the non-hand-labelled markers to obtain their image on the target geometry
Mesh fitting stage: Finding the optimal distribution of
control points that:
a) Minimizes the “distance” between the reference face deformed by the retargeted control mesh and the target geometry
b) Preserves the general shape represented by a deformation energy function
c) Stays on the surface of the target face (enforced through the cylindrical mapping)
Simplex downhill method
4.2. Scaling Facial Motion
4.2. Scaling Facial Motion The two faces are labelled with
the same markers
After fitting the control mesh
We can extend this mapping to the whole space the faces are given in
By interpolating the initial displacement at every control point using RBFs
This interpolant is used to compute the mapping on the target space of the captured markers during the animation
4.2. Scaling Facial Motion
An evaluation the 2-norm of the metric tensor of the mapping shows how infinitesimal displacements are scaled
green: positive scaling (>1)
blue: negative scaling
The Planar Bones algorithm computes the final deformation, driven by the retargeted control mesh
This procedure implicitly scales the movement of the markers in the target space, according to the initial correspondence that is given as reference.
5. Processing Motion Capture Input
Limitations in the marker tracking technology lead to deficiencies in the captured data:
6. Sample results: lip tracking
6. Sample results: lip tracking
7. Conclusions and future work We have introduced a novel method for the retargeting
and animation of faces from motion capture data
Current research: Provide a better model for the tracking of the inner contour of the
lips: Marker-less image processing of the video capture Physical model using a mass-spring system attached to the outer
contour
Introduce furrowing and wrinkling in the skin animation A posteriori deformation analysis on the deformation induced by Planar
Bones
Questions?
m.sanchez @dcs.shef.ac.ukj.edge @dcs.shef.ac.uk
Additional samples: chorus
Top Related