Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa...
-
Upload
francis-ryan -
Category
Documents
-
view
218 -
download
0
Transcript of Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa...
![Page 1: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/1.jpg)
Graphite 2004Statistical Synthesis of Facial
Expressions for the Portrayal of Emotion
Lisa GralewskiBristol UniversityUnited [email protected]
![Page 2: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/2.jpg)
Aim and Motivation:
Generate new emotion specific video sequences.
Facial expressions + head motion. Potentially infinitely long No repeated footage
Provide an animation tool: User driven. Allows emotion selection. Automatic generation of novel facial expression
sequences.
![Page 3: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/3.jpg)
Approach: Statistical Approach
Divides into 2 stages
Stage 1 Modelling: To form generative emotion models.
• Shape model •point data, potential to drive animations
• Combined appearance space model•texture and shape for the generation of new video frames.
Stage 2 Synthesis: Method to generate new video footage / point data
![Page 4: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/4.jpg)
Emotion video footage is considered as a time series of changing facial expressions.
Time series can be modelled with Auto regressive processes (ARPs) Advantages:
constructed models generate new data similar to training data. Potentially infinitely long generated sequences.
Disadvantages: For successful modelling, input data must have stationary signal.
Use ARP’s to form generative emotion specific models. These allow synthesis of new facial expression emotion sequences.
Stage 1: Modelling Approach
Time
![Page 5: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/5.jpg)
Modelling: Data Collection
Acquire video footage Emotions: Happy, sad
and, angry.
Split video into:
1) Shape representation Using hand labelled
control points.
2) Texture representation Using warped image set
Neutral pose mean shape of control point set.
![Page 6: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/6.jpg)
Modelling: Facial expressions
Apply principle components analysis (PCA) video data to calculate eigenvectors for:
Shape space (point data) Texture space (warped pixel data) Combined space (shape + texture).
Calculate shape and combined space responses.
Data projected from a higher-D space down to lower D-space. Each video frame has a response Together the responses for a sequence form an eigensignature.
Form linear shape and combined space models. Allow reconstruction of original data.
![Page 7: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/7.jpg)
Modelling: Eigensignatures Visualise eigensignature:
Plot of 1st and 2nd principle components (responses).
Each point of the eigensignature represents a single video frame.
AngryHappy
Sad
SHAPE EIGENSIGNATURE
Happy
Sad
Angry
COMBINED EIGENSIGNATURE
![Page 8: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/8.jpg)
Modelling: Construct ARP’s
3 video clips yield: Happy eigensignature Sad eigensignature Angry eigensignature
Consider eigensignature as a multivariate time series.
Separate application of multivariate-ARP’s Results: separate ARP emotion models.
Models used to generate new emotion specific eigensignatures-(stage 2: SYNTHESIS)
![Page 9: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/9.jpg)
Stage 2: Synthesis
Generate eigensignatures Potentially infinitely long Retains look and feel of original.
Original Eigensignature: Emotion Angry
Generated ARPEigensignature: Emotion Angry
![Page 10: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/10.jpg)
Synthesis: Generated Shape ARP sequences:
Original Angry ARP Generated Angry
Using linear shape space PCA models.Project generated shape eigensignature back to shape space.
![Page 11: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/11.jpg)
Synthesis: Generated Shape ARP sequences:
ARP Generated Happy ARP Generated Sad
![Page 12: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/12.jpg)
Synthesis: Generated Combined space ARP sequences:
Original Angry ARP Generated Angry
Using combined appearance PCA models. Project generated eigensignatures back to combined appearance
space.
![Page 13: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/13.jpg)
Synthesis: Generated Combined space ARP sequences:
ARP Generated Happy ARP Generated Sad
![Page 14: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/14.jpg)
Application: User-driven synthesis
Approach: Self organising maps (SOMs) SOM’s project the n-D ARP emotion data into a 2D
image space, defined as ‘expression space’.
Tool development Expression Space visualises emotion ARP models. Navigate space. Automatic generation new emotion specific videos.
![Page 15: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/15.jpg)
Self-Organising Maps
Clustering technique Kohonen Neural Network Visualises n-observations as
g-groups.
SOM splits into 2 elements1) Data – provided by ARP
models Requires sufficient training
data.
2) Weight vectors components. xy position on map After training weight vectors
equivalent to ARP models.
x1,y1
W1, W2 W3 W4
Weight Vectors ARP-coefficients
x2,y2 x3,y3 x4,y4
SOM UNITS
Self-Organising Map Structure
![Page 16: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/16.jpg)
Technique: Acquisition of SOM training data.
COMPLETE EMOTION EIGENSIGNATURE
• Use N emotion ARP’s to train 50x50 unit SOM.
Split complete eigensignature into frame segments with (length-1)
frame overlap.
1 2 N-segments
• Model each segment with an ARP.• Result: obtain N ARP models for SOM training.
Results After training :•Each unit of the SOM is equivalent to a single ARP model.•ARP emotion models are clustered.
![Page 17: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/17.jpg)
Technique: Forming Expression Space
Generate Expression space by colour coding the SOM units. Use an Euclidean distance scheme, based on comparison between the sample ARP
emotion models used for training and SOM map unit ARP’s.
Red units most similar to angry ARP models
Green units most similar
to happy
Blue unit associate to
sad ARP models
Intensity of colour indicates the better match to an emotion group.
![Page 18: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/18.jpg)
Results: SOM Shape Space SOM Combined Space SOM
Emotions cluster into 3 distinct regions. Distinct separation between angry and all other emotions. Large transitional region between sad and happy. Dark regions not closely related to any emotion.
Angry Cluster
Sad Cluster
Happy Cluster
Angry Cluster
Sad Cluster
Happy Cluster
![Page 19: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/19.jpg)
Navigating Expression Space
This tool provides an intuitive interface.
Aimed at non-specialist users
Allows easy traversal of the highly dimensional expression space.
Automatically generates new facial expression sequences for selected emotions.
SOMVisualisation
Generated Expression
![Page 20: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/20.jpg)
Navigating the Combined Appearance Space
SOMVisualisation
GeneratedVideo texture
Complete Eigensignature
Generated Eigensignature
![Page 21: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/21.jpg)
Results: Using different ARP models from the SOM to generate sad crowd.
![Page 22: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/22.jpg)
Mixed Crowd Scene
![Page 23: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/23.jpg)
Conclusions
Demonstrated approach which can reconstruct and generate novel footage.
Successful use of PCA and ARP’s: To construct emotion models that correctly co-
articulate head motion and facial expression.
Eigensignature representation unique for each emotion. Emotion ARP’s cluster distinctly Clustered space easily navigable by non-
specialists.
![Page 24: Graphite 2004 Statistical Synthesis of Facial Expressions for the Portrayal of Emotion Lisa Gralewski Bristol University United Kingdom gralewsk@cs.bris.ac.uk.](https://reader035.fdocuments.us/reader035/viewer/2022062408/56649e845503460f94b85ade/html5/thumbnails/24.jpg)
Finally……
Any questions?