Project 10Facial Emotion Recognition Based On Mouth Analysis

SSIP 08, Vienna1

http://www.we-hope-project10-will-win.info

The Project

Objective : To recognize emotional state / expression using mouth information

Input: Mouth images (no make-up)

Output: Emotional State/ Expression Happy, Neutral, Sad

2

http://www.we-hope-project10-will-win.info

The Team

3

Kornélprogrammer

Kornélprogrammer

PéterWeb programmer

PéterWeb programmer

Kamalprogrammer

Kamalprogrammer

Naiemresearcher

Naiemresearcher

Sofiaprogrammer

Sofiaprogrammer

http://www.we-hope-project10-will-win.info

The Tasks

Create facial expressions photographic database

Segment the mouth in the input image

Use suitable features for expression characterization

Design a reliable classifier to distinguish between different mouth expressions

4

http://www.we-hope-project10-will-win.info

SSIP Lips database

Happy, Neutral and Sad Photos of SSIP students and lecturers

Thank you all!!!

Happy Neutral Sad

5

http://www.we-hope-project10-will-win.info

Mouth Segmentation

6

Input Image HSV Space - Hue Thresholding

Morphological Operations

http://www.we-hope-project10-will-win.info

Segmentation Results…

And Segmentation Problems…

7

http://www.we-hope-project10-will-win.info

Lips Features Extraction

Detect the leftmost and rightmost lip points

Normalize images (rotation, translation and scaling)

Calculate features Eccentricity Convex Area Minor Axis Ratio of Upper to Lower Lip

8

http://www.we-hope-project10-will-win.info

Expression Classification

SVM Classifier

Two Stage Classification

Mouth Features

☺

9

http://www.we-hope-project10-will-win.info

Results 1

Differences between different classes were found to be statistically significant (p<0.01)

Classification Accuracy Stage 1 (Sad / Not Sad) 88% Stage 2 (Happy/ Neutral) 62%

10

http://www.we-hope-project10-will-win.info

Results 2

11

http://www.we-hope-project10-will-win.info

Future Work Acquire larger database for training and testing

Test different facial expressions (such as anger and disgust)

Other classifiers: NN, FIS

Conclusion Mouth information is often insufficient for

recognizing facial expression / emotional state

Other face features such as eyes and eyebrows can contribute in emotional state recognition

12

http://www.we-hope-project10-will-win.info

GUI

13

http://www.we-hope-project10-will-win.info

References M. Gordan, C. Kotropoulos, I. Pitas, “Pseudoautomatic Lip Contour Detection Based on Edge

Direction Patterns”

J. Kim, S. Na, R. Cole, “Lip Detection Using Confidence-Based Adaptive Thresholding”

F. Tang, “Facial Expression Recognition using AAM and Local Facial Features”

M. Pantic, M. Tomc, L. Rothkrantz , “A Hybrid Approcah to Mouth Features Detection”

14

http://www.we-hope-project10-will-win.info

Thank you for your attention!!!

15

http://www.we-hope-project10-will-win.info