Detection and Recognition of Facial Emotionusing Bezier Curves

Yong-Hwan LeeDankook University

152, Jukjeon-ro, Suji-gu, Yongin-si, Gyeonggi-do, 448-701, KoreaTel: +82-10-3072-1458

hwany1458@empal.com

Abstract

Extracting and understanding of emotion is of high importance for the interaction among humanand machine communication systems. The most expressive way to display the human’s emotion isthrough facial expression analysis. This paper presents and implements an automatic extraction andrecognition method of facial expression and emotion from still image. There are two steps to rec-ognize the facial emotion; (1) Detecting facial region with color feature-map, and (2) Verifying theemotion of characteristic with Bezier curve. To evaluate the performance of the proposed algorithm,we assess the ratio of success with emotionally expressive facial image database. Experimental re-sults shows average 78.8% of success to analyze and recognize the facial expression and emotion.The obtained result indicates the good performance and enough to applicable to mobile environments.

Keywords: Emotion Recognition, Face Recognition, Facial Expression Analysis, Bezier Curve.

1 Introduction

Recognition and analysis of human facial expression and emotion have attracted a lot of interest inthe past few decades, and they have been researched extensively in neuroscience, cognitive sciences,computer sciences and engineering [2]. These researches focus not only on improving human-computerinterfaces, but also on improving the actions which computer takes on feedback by the user. Whilefeedback from the user has traditionally been occurred by using keyboard or mouse, in the recent, smart-phone and camera enable the system to see and watch the user’s activities, and this leads that the user caneasily utilize intelligent interaction. Human interact with each other not only through speech, but alsothrough gestures, to emphasize a certain part of the speech, and to display of emotions. Emotions of theuser are displayed by visual, vocal and other physiological means [3]. There are many ways to displayhuman’s emotion, and the most natural way to display emotions is using facial expressions, which aremostly based on video sequences [17].

This paper proposes a scheme to automatically segment an input still image, and to recognize facialemotion using detection of color-based facial feature map and classification of emotion with simple curveand distance measure is proposed and implemented. The motivation of this paper is to study the effect offacial landmark, and to implement an efficient recognition algorithm of facial emotion with still image,while most of researches are using video sequences due to utilize the differences between frames.

This paper is organized as follows: next section reviews related works about emotion recognitionthrough facial expression analysis. Section 3 presents the proposed system that uses two main step torecognize and classify the facial emotion from still image. Experimental results are shown in section 4,and section 5 summarizes the study and its future works.

IT CoNvergence PRActice (INPRA), volume: 1, number: 2, pp. 11-19

11

Detection and Recognition of Facial Emotion Yong-Hwan

2 Related Works

A number of recently papers exist on automatic affect analyze and recognition of human emotion [6].Research on recognizing emotion through facial expression was pioneered by Ekman [4], who startedtheir work from the psychology perspective. Ira et al [3] proposed an architecture of Hidden MarkovModels for automatically segment and recognize human facial expression from video sequences. Yash-nari [10] investigated a method for facial expression recognition for a human speaker by using thermalimage processing and a speech recognition system. He improved speech recognition system to savethermal images, just before and just when speaking the phonemes of the first and last vowels, throughintentional facial expressions of five categories with emotion. Spiros et al [8] suggested the extraction ofappropriate facial features and consequent recognition of the user’s emotional state that could be robustto facial expression variations among different users. They extracted facial animation parameters definedaccording to the ISO MPEG-4 standard by appropriate confidence measures of the estimation accuracy.But, most of these research are used a method to recognize emotion based on videos using extractingframes. Several prototype systems were published that can recognize deliberately produced action unitsin either frontal view face images [14] or profile view face images [13]. These systems employ differentapproaches including expert rules and machine learning methods such as neural networks, and use eitherfeature-based image representation or appearance-based image representation. Valstar et al [19] employsprobabilistic, statistical and ensemble learning techniques, which seem to be particularly suitable forautomatic action unit recognition from face image sequences.

3 Proposed Method

The proposed method for recognition of facial expression and emotion is composed of two major steps:first one is a detecting and analysis of facial area from original input image, and next is a verification ofthe facial emotion of characteristic features in the region of interest. A block diagram of our proposedmethod is depicted in Figure 1.

Figure 1: The proposed system flowchart

In the first step for face detection, the proposed method locates and detects a face in a color stillimage based on the skin color and the region of eye and mouth. Thus, the algorithm first extract the skincolor pixels by initialized spatial filtering, based on the result from the lighting compensation. Then,the method estimates a face position and the region of facial location for eye and mouth by feature map.After obtaining the region of interest, we extract points of the feature map to apply Bezier curve on eyeand mouth.

12

Detection and Recognition of Facial Emotion Yong-Hwan

Then, understanding and recognition of facial emotion is performed by training and measuring thedifference of Hausdorff distance with Bezier curve between input face image and facial images in thedatabase.

3.1 Detection of Skin Color using YCbCr

A first step of our scheme is color space transformation and lighting compensation. Although skin colorappears to vary, we assume that there exists underlying similarities in the chromatic properties of allfaces and that all major differences lie in intensity rather than in the facial skin color itself. In this case,we have adopted to utilize a skin-color based approach using YCbCr color model. With the color model,luminance information is represented by a single component, Y, and color information is stored as twocolor difference component, Cb and Cr [5].

Y = 0+(0.299 ·R)+(0.587 ·G)+(0.114 ·B)Cb = 128− (0.168736 ·R)− (0.331264 ·G)+(0.5 ·B)Cr = 128+(0.5 ·R)− (0.418688 ·G)− (0.081312 ·B)

(1)

After converted color model, an illumination calibration is pre-requisite during the pre-processingfor the accurate face detection. Since the illumination condition is an important factor to effect on theperformance of detection, we attempt pre-processing to equalize the intensity value in an image as follow;

Y ′ = (y−min1

max1−min1)(max2−min2)+min2 , i f (y≤ Kl or Kh ≤ y) (2)

where min1 and max1 are minimum and maximum value of Y component on input image, min2 andmax2 are the value of the transformed space, Kl = 30 and Kh = 220. The values of these experientialparameters are estimated from training data sets of skin patches of sample database.

Histogram equalization enhances the performance which brightness is second into one direction,

Pk(rk) =nk

n, 0≤ rk ≤ 1 , k = 0,1, · · · , l−1 (3)

where l is the number of discrete values for the intensity, n is the number of total pixels in the image,rk is the k th intensity, and nk is the number of pixels which intensity is rk. Frequency cumulativeness isdependent on rk, thus, following Equation (3) can be used for the intensity equalization.

sk =k

∑j=0

nk

n= E(rk) =

k

∑j=0

Pr(r j) (4)

3.2 Eye and Mouth Detection using Feature Map

After reducing the illumination impact to the brightness component, we attempt to extract the region ofeye and mouth on an image using Eyemap and Mouthmap in Equation (5) and Equation (6) [11]. Regionof eyes would be easily found due to its intrinsic feature, namely symmetry. This paper restricts thatboth both eyes should be present inside the image to detect the skin region. As observed in [7], eyes arecharacterized by low red component and high blue one in the CbCr planes. Thus, Eyemap transformationis constructed by:

EyeMap =13(α · (Cb)

2 +β · (Cr)2 +(

Cb

Cr)) (5)

13

Detection and Recognition of Facial Emotion Yong-Hwan

where (Cb)2 , Cr)

2 and CbCr

all are normalized to the range [0,255] and Cr is the negative of Cr, and α

is greater than 1, β is less than 1 of positive constant which emphasizes to increase or decrease the redand blue component, because of Asian skin color, which generally have R≥ G≥ B pattern.

Mouth is characterized by a high red component and low blue one, thus the mouth region has arelatively high response in the (Cb)

2 feature, low response in the CrCb

. Mouthmap is constructed by followEquation (6).

MouthMap = C2r × (Cr− (

α ·Cr

255 ·Cb)) (6)

where α = 0.80, which is the experiential value.

Figure 2: Detection of face boundary and features using Eyemap and Mouthmap; (a) Original inputimage, (b) Skin-tone extracted image, (c) Eyemap, (d) Mouthmap, and (e) Face boundary map

3.3 Drawing Bezier Curve on Eye and Mouth

The Bezier curve generates contour points considering global shape information with the curve passingthrough the first and last control points [16]. If there are L+ 1 control points, the position is defined asPk : (xk,yk),0 ≤ k ≤ L considering 2D shapes. These coordinate points are then blended to form P(t),which describes the path of Bezier polynomial function between P0 and PL:

P(t) =L

∑k=0

PkBEZk,L(t) (7)

where the Bezier blending function BEZk,L(t) is known as the Bernstein polynomial, which is definedas [15]:

BEZk,L(t) = (Lk)tk(1− t)L−k (8)

The recursive formula which are used to decide coordinate locations is obtained by:

BEZk,L(t) = (1− t) ·BEZk,L−1(t)+ t ·BEZk,L−1 + t ·BEZk−1,L−1(t) (9)

where BEZk,k(t) = tk and BEZ0,k(t) = (1− t)k.The coordinates of individual Bezier curve are represented by the following pair of parametric equa-

tions:

x(t) = ∑Lk=0 xkBEZk,L(t)

y(t) = ∑Lk=0 ykBEZk,L(t)

(10)

An example of a Bezier curve is given in Figure 3 with 4 control points [1].

14

Detection and Recognition of Facial Emotion Yong-Hwan

Figure 3: Examples of construction of the Bezier curve

For applying the Bezier curve, we need to extract some control points of each interest regions, wherelocate in the area of left eye, right eye and mouth. Thus, we apply big connect for finding the high-est connected area within each interest regions from the eyemap and mouthmap. Then, we find fourboundary points of the regions, which are the starting and ending pixels in horizontals, and the top andbottom pixels of the central points in verticals. After getting four boundary points of each regions, theBezier curves for left eye, right eye and mouth are obtained by drawing tangents to a curve over the fourboundary control points.

3.4 Training and Recognizing Facial Emotion with Hausdorff Distance

In training database, there are two tables which are storing for personal information and indexes of fouremotions with his/her own curves of facial expression analysis. For detection of facial emotion, we needto compute the distance a one-to-one correspondence of each interest regions between an input imageand the images in the database. The Bezier curves are drawn over principal lines of facial features.To estimate a similarity matching, we first normalize the displacements that converts each width ofthe Bezier curve to 100 and height according to its width. We then apply the Hausdorff distance tocompare the shape metric between them. The distance dH(p,q) between two curves p(s),s ∈ [a,b] andq(t), t ∈ [c,d] is given in Equation (10) [9].

dH(p,q) = max{maxs∈[a,b]

mint∈[c,d]

|p(s)−q(t)|, maxt∈[c,d]

mins∈[a,b]

|p(s)−q(t)|} (11)

4 Experiments and Results

The expressions such as smile, sad, surprise and neutral are considered for the experiment of face recog-nition. The faces with expressions are compared against the model face database consisting of neutralfaces. All the face images are normalized using some parameters. The Bezier points are interpolated overthe principal lines of facial features. These points for each curve form the adjacent curve segments. TheHausdroff distance is calculated based on the curve segments. Then, understanding and decision of facialemotion is chosen by measuring similarity in faces. The ground truth set for estimating the performanceof the algorithm is provided with the categories in the experiments, which are correct if the decision isbelonged to the correct category. Figure 5 shows how to move the control points across different subjects(e.g., neutral and smile) and to interpret a tracking facial features with Bezier curve and extracted featurepoints interpolation.

To categorize facial emotion, we need first to determine the expressions from movements of facialcontrol points. Ekman et al. [4] have produced a system for describeing visually distinguishable facialmovements (called Facial Action Coding System, FACS), which is based on enumeration of all action

15

Detection and Recognition of Facial Emotion Yong-Hwan

Figure 4: Bezier curve and extracted feature points interpolation across different subjects. (a) line fittingover example image in neutral condition, and (b) in smile condition

Table 1: Rule of facial emotion recognition

Emotions Movements of AUsSmile Eye opening is narrowed,

Mouth is opening, and Lip corners are pulled obliquelySad Eye is tightened closed, and

and Lower lip corner depressorSurprise Eye and Mouth are opened, Upper eyelid raiser,

and Mouth stretch

units(AUs). There are 46 AUs in FACS that account for changes in facial expression, and combinationrules of the AUs are considered to determine form defining emotion-specified expressions. The rules forfacial emotion are created by basic AUs from FACS, and the decision of recognition is determined fromtheir rules, as given in Table 1.

In this work, the facial expressions have been recognized only by static image. Testing of the al-gorithm is performed on a database of people, which is obtained from FEI face database [18] andJAFFE database [12]. There are 14 images for each of 200 individuals, total of 2,800 images in FEIface database. All images are colorful and taken against a white homogenous background in an uprightfrontal position. Additionally, they provide a subset of facial images, totally 400 images, each subject hastwo frontal images (one with neutral and the other with smiling facial expression). JAFFE database con-tains 213 images of 7 facial expressions posed by 10 Japanese female models. Each image has been ratedon 6 emotion adjectives by 60 Japanese subjects. We used a subset of facial images from two databasewhich consisted of 250 images and four categories in neutral, smile, sad and surprise. Figure 5 showssample facial images which expresses an emotion, for example, by neutral and smiling expression. Thealgorithm presented in previous section is implemented with visual C#, and experiments are performedon a Intel Core 3.1 GHz PC with 4 GB RAM, as shown in Figure 6.

The experiments shows the recognition results under different facial expressions such as smile, sad,surprise and neutral. The proposed method recognizes 197 of the 250 faces, which means that a suc-cessful recognition ratio of 78.8% is achieved, as shown in Table 2. Experimental result reveals thatsuccess ratio is better for smile, because the control points and curves for smile are given more apprecia-ble changes from neutral expression. Among the 53 false dismissals cases, some of faces are gray-scaleimages from JAFFE database, which leads hard to detect the skin color pattern of facial region in theimage.

16

Detection and Recognition of Facial Emotion Yong-Hwan

Figure 5: Examples of images for three subjects from sample database: Neutral facial emotion at firstrow, and smiling for happy at the bottom

Figure 6: Screenshot of the implemented system: face recognition and analysis for facial emotion

5 Conclusion

In this paper, we have presented and implemented a simple approach for recognition of the facial ex-pression analysis. The algorithm is performed two major steps: one is a detection of facial region withskin color segmentation and calculation of feature-map for extracting two interest regions focused oneye and mouth. And the other is a verification of the facial emotion of characteristic features with theBezier curve and the Hausdorff distance. Experimental results shows average successful ratio of 78.8%to recognize the facial expression, and this indicates the good performance and enough to applicable tomobile devices.

Table 2: Results of facial expression recognition

Expression Corrects / Misses Success RatioSmile 67 / 12 84.8%Sad 22 / 8 73.3%

Surprise 45 / 16 73.8%Neutral 63 / 17 78.8%Total 197 / 53 78.8%

17

Detection and Recognition of Facial Emotion Yong-Hwan

References[1] Bezier curve. Wikipedia, 2013. http://en.wikipedia.org/wiki/Bezier_curve.[2] R. Adolphs. Recognizing Emotion From Facial Expressions: Psychological and Neurological Mechanisms.

Behavioral and Cognitive Neuroscience Reviews, 2002.[3] I. Cohen, A. Garg, and T. S. Huang. Emotion Recognition from Facial Expressions using Multilevel HMM.

In Proc. of the 2000 NIPS Workshop on Affective Computing, Denver, Colorado, USA, December 2000.[4] P. Ekman and W. Friesen. Facial Action Coding System (FACS): Investigator’s Guide. Conculting Psychol-

ogists Press, 1978.[5] R. C. Gonzalea, R. E. Woods, and S. L. Eddins. Digital Image Processing using Matlab. Prentice Hall, 2004.[6] G. H. and P. M. Automatic Temporal Segment Detection and Affect Recognition from Face and Body

Display. IEEE Transactions on Systems, Man, and Cybernetics - Part B, 39(1):64–84, 2009.[7] R.-L. Hsu, M. Abdel-Mattaleb, and A. K. Jain. Face Detection in Color Images. IEEE Transactions on

Pattern Analysis and Machine Intelligence, 24(5):696–701, 2002.[8] S. V. Ioannou, A. T. Raouzaiou, V. A. Tzouvaras, T. P. Mailis, K. C. Karpouzis, and S. D. Kollias. Emo-

tion Recognition through Facial Expression Analysis based on a Neurofuzzy Network. Neural Networks,18(4):423–435, 2005.

[9] S.-H. Kim and Y. J. Ahn. An Approximation of Circular Arcs by Quartic Bezier Curves. Computer-AidedDesign, 39(6):490–493, 2007.

[10] Y. Koda, Y. Yoshitomi, M. Nakano, and M. Tabuse. Facial Expression Recognition for Speaker using ThermalImage Processing and Speech Recognition System. In Proc. of the 18th IEEE International Symposiumon Robot and Human Interactive Communication (RO-MAN’09), Toyama, Japan, pages 955–960. IEEE,September-October 2009.

[11] Y.-H. Lee, T.-K. Han, Y.-S. Kim, and S.-B. Rhee. An Efficient Algorithm for Face Recognition in MobileEnvironments. Asian Journal of Information Technology, 4(8):796–802, 2005.

[12] M. J. Lyons, M. Kamachi, and J. Gyoba. Japanese Femail Facial Expressions (JAFFE). Database of digitalimages, 1997.

[13] P. M. and I. Patras. Dynamics of facial expression: Recognition of facial actions and their temporal segmentsfrom face profile image sequences. IEEE Transactions on Systems, Man and Cybernetics, 36(2):433–449,2006.

[14] P. M. and L. Rothkrantz. Facial Action Recognition for Facial Expression Analysis from Static Face Images.IEEE Transactions on Systems, Man and Cybernetics - Part B, 34(3):1449–1461, 2004.

[15] M. R. Rahman, M. A. Ali, and G. Sorwar. Finding Significant Points for Parametric Curve GenerationTechniques. Journal of Advanced Computations, 2(2), 2008.

[16] B. Sam and B. R. Samuel. 3D Computer Graphics - A Mathematical Introduction with OpenGL. CambridgeUniversity Press, 2003.

[17] N. Sebe, I. Cohen, T. Gevers, and T. S. Huang. Multimodal Approaches for Emotion Recognition: A Survey.In S. Santini, R. Schettini, and T. Gevers, editors, Internet Imaging VI, SPIE, volume 5670, pages 56–67.Society of Photo Optical, January 2005.

[18] C. Thomaz and G. Giraldi. A New Ranking Method for Principal Components Analysis and its Applicationto Face Image Analysis. Image and Vision Computing, 28(6):902–913, 2010.

[19] M. Valster, P. M., Z. Ambadar, and J. Cohn. Spontaneous vs. Posed Facial Behavior: Automatic Analysis ofBrow Actions. pages 162–170. ACM, November 2006.

18

Detection and Recognition of Facial Emotion Yong-Hwan

Author BiographyYong-Hwan Lee received the M.S. degree in Computer Science and Ph.D. in Elec-tronics and Computer Engineering from Dankook University, Korea, in 1995 and2007, respectively. Currently, he is a research professor at the Department of Ap-plied Computer Engineering, Dankook University, Korea. His research areas includeImage/Video Representation and Retrieval, Face Recognition, Augmented Reality,Mobile Programming and Multimedia Communication.

19