Automatic Recognition of Facial Actions in Spontaneous ...€¦ · The facial action coding system...

Automatic Recognition of Facial Actions inSpontaneous Expressions

Marian Stewart Bartlett1, Gwen C. Littlewort1, Mark G. Frank2, Claudia Lainscsek1,Ian R. Fasel1, Javier R. Movellan1

1Institute for Neural Computation, University of California, San [email protected], [email protected], [email protected], [email protected],

[email protected]

2Department of Communication, University at Buffalo, StateUniversity of New [email protected]

Abstract— Spontaneous facial expressions differ from posedexpressions in both which muscles are moved, and in the dy-namics of the movement. Advances in the field of automaticfacial expression measurement will require developmentand assessment on spontaneous behavior. Here we presentpreliminary results on a task of facial action detection inspontaneous facial expressions. We employ a user indepen-dent fully automatic system for real time recognition offacial actions from the Facial Action Coding System (FACS).The system automatically detects frontal faces in the videostream and coded each frame with respect to 20 Actionunits. The approach applies machine learning methods suchas support vector machines and AdaBoost, to texture-basedimage representations. The output margin for the learnedclassifiers predicts action unit intensity. Frame-by-frameintensity measurements will enable investigations into facialexpression dynamics which were previously intractable byhuman coding.

I. I NTRODUCTION

A. The facial action coding system

In order to objectively capture the richness and com-plexity of facial expressions, behavioral scientists havefound it necessary to develop objective coding standards.The facial action coding system (FACS) [17] is the mostwidely used expression coding system in the behavioralsciences. A human coder decomposes facial expressionsin terms of 46 component movements, which roughlycorrespond to the individual facial muscles. An exampleis shown in Figure 1.

FACS provides an objective and comprehensive lan-guage for describing facial expressions and relating themback to what is known about their meaning from thebehavioral science literature. Because it is comprehensive,FACS also allows for the discovery of new patterns relatedto emotional or situational states. For example, what arethe facial behaviors associated with driver fatigue? Whatare the facial behaviors associated with states that arecritical for automated tutoring systems, such as interest,boredom, confusion, or comprehension? Without an ob-jective facial measurement system, we have a chicken-

and-egg problem. How do we build systems to detectcomprehension, for example, when we don’t know forcertain what faces do when students are comprehending?Having subjects pose states such as comprehension andconfusion is of limited use since there is a great deal ofevidence that people do different things with their faceswhen posing versus during a spontaneous experience (e.g.[8], [14]). Likewise, subjective labeling of expressionshas also been shown to be less reliable than objectivecoding for finding relationships between facial expressionand other state variables. Some examples of this arediscussed below, namely the failure of subjective labelsto show associations between smiling and other measuresof happiness, as well as failure of naive subjects to differ-entiate deception and intoxication from facial expression,whereas reliable differences were shown with FACS.

Objective coding with FACS is one approach to theproblem of developing detectors for state variables suchas comprehension and confusion, although not the onlyone. Machine learning of classifiers from a databaseof spontaneous examples of subjects in these states isanother viable approach, although this carries with itissues of eliciting the state, and assessment of whetherand to what degree the subject is experiencing the desiredstate. Experiments using FACS face the same challenge,although computer scientists can take advantage of a largebody of literature in which this has already been done bybehavioral scientists. Once a database exists, however, inwhich a state has been elicited, machine learning can beapplied either directly to image primitives, or to facialaction codes. It is an open question whether intermediaterepresentations such as FACS are the best approach torecognition, and such questions can begin to be addressedwith databases such as the one described in this paper.Regardless of which approach is more effective, FACSprovides a general purpose representation that can beuseful for many applications. It would be time consumingto collect a new database and train application-specific de-tectors directly from image primitives for each new appli-

22 JOURNAL OF MULTIMEDIA, VOL. 1, NO. 6, SEPTEMBER 2006

© 2006 ACADEMY PUBLISHER

cation. The speech recognition community has convergedon a strategy that combines intermediate representationsfrom phoneme detectors plus context-dependent featurestrained directly from the signal primitives, and perhapsa similar strategy will be effective for automatic facialexpression recognition.

There are numerous examples in the behavioral scienceliterature where FACS enabled discovery of new relation-ships between facial movement and internal state. Forexample, early studies of smiling focused on subjectivejudgments of happiness, or on just the mouth movement(zygomatic major). These studies were unable to show areliable relationship between expression and other mea-sures of enjoyment, and it was not until experiments withFACS measured facial expressions more comprehensively,that a strong relationship was found: Namely that smileswhich featured both orbicularis oculi (AU6), as well as zy-gomatic major action (AU12), were correlated with self-reports of enjoyment, as well as different patterns of brainactivity, whereas smiles that featured only zygomaticmajor (AU12) were not (e.g. [16]). Research based uponFACS has also shown that facial actions can show differ-ences between genuine and faked pain [8], and betweenthose telling the truth and lying at a much higher accuracylevel than naive subjects making subjective judgments ofthe same faces [26]. Facial Actions can predict the onsetand remission of depression, schizophrenia, and otherpsychopathology [20], can discriminate suicidally fromnon-suicidally depressed patients [27], and can predicttransient myocardial ischemia in coronary patients [42].FACS has also been able to identify patterns of facialactivity involved in alcohol intoxication that observers nottrained in FACS failed to note [44].

Figure 1. Example FACS codes for a prototypical expression of fear.Spontaneous expressions may contain only a subset of these ActionUnits.

Although FACS has a proven record for the scien-tific analysis of facial behavior, the process of applyingFACS to videotaped behavior is currently done by handand has been identified as one of the main obstaclesto doing research on emotion [15], [25]. FACS codingis currently performed by trained experts who makeperceptual judgments of video sequences, often frameby frame. It requires approximately 100 hours to traina person to make these judgments reliably and pass astandardized test for reliability. It then typically takesover two hours to code comprehensively one minute ofvideo. Furthermore, although humans can be trained to

code reliably the morphology of facial expressions (whichmuscles are active) it is very difficult for them to code thedynamics of the expression (the activation and movementpatterns of the muscles as a function of time). There isgood evidence suggesting that such expression dynamics,not just morphology, may provide important information[18]. For example, spontaneous expressions have a fastand smooth onset, with distinct facial actions peakingsimultaneously, whereas posed expressions tend to haveslow and jerky onsets, and the actions typically do notpeak simultaneously [24].

Significant advances in computer vision open up thepossibility of automatic coding of facial expressions atthe level of detail required for such behavioral studies.Automated systems would have a tremendous impact onbasic research by making facial expression measurementmore accessible as a behavioral measure, and by providingdata on the dynamics of facial behavior at a resolutionthat was previously unavailable. Such systems would alsolay the foundations for computers that can understandthis critical aspect of human communication. Computersystems with this capability have a wide range of ap-plications in basic and applied research areas, includingman-machine communication, security, law enforcement,psychiatry, education, and telecommunications [39].

B. Spontaneous Facial Expression

The importance of spontaneous behavior for developingand testing computer vision systems becomes apparentwhen we examine the neurological substrate for facial ex-pression. There are two distinct neural pathways that me-diate facial expressions, each one originating in a differentarea of the brain. Volitional facial movements originatein the cortical motor strip, whereas the more involuntary,emotional facial actions, originate in the subcortical areasof the brain (e.g. [33]. Research documenting these dif-ferences was sufficiently reliable to become the primarydiagnostic criteria for certain brain lesions prior to modernimaging methods (e.g. [6].) The facial expressions medi-ated by these two pathways have differences both in whichfacial muscles are moved and in their dynamics. The twoneural pathways innervate different facial muscles [41],and there are related differences in which muscles aremoved when subjects are asked to pose an expression suchas fear versus when it is displayed spontaneously [14].Subcortically initiated facial expressions (the involuntarygroup) are characterized by synchronized, smooth, sym-metrical, consistent, and reflex-like facial muscle move-ments whereas cortically initiated facial expressions aresubject to volitional real-time control and tend to be lesssmooth, with more variable dynamics (see review by Rinn[40].) However, precise characterization of spontaneousexpression dynamics has been slowed down by the needto use non-invasive technologies (e.g. video), and thedifficulty of manually coding expression intensity frame-by-frame. Thus the importance of video based automaticcoding systems.

JOURNAL OF MULTIMEDIA, VOL. 1, NO. 6, SEPTEMBER 2006 23


These two pathways appear to correspond to thedistinction between biologically driven versus sociallylearned facial behavior. Researchers agree, for the mostpart, that most types of facial expressions are learnedlike language, displayed under conscious control, andhave culturally specific meanings that rely on context forproper interpretation (e.g. [13]). Thus, the same loweredeyebrow expression that would convey ”uncertainty” inNorth America might convey ”no” in Borneo [9]. On theother hand, there are a limited number of distinct facialexpressions of emotion that appear to be biologicallywired, produced involuntarily, and whose meanings aresimilar across all cultures; for example, anger, contempt,disgust, fear, happiness, sadness, and surprise [13]. Anumber of studies have documented the relationship be-tween these facial expressions of emotion and the phys-iology of the emotional response (e.g. [19], [20].) Thereare also spontaneous facial movements that accompanyspeech. These movements are smooth and ballistic, andare more typical of the subcortical system associatedwith spontaneous expressions (e.g. [40]). There is someevidence that arm-reaching movements transfer from onemotor system when they require planning to another whenthey become automatic, with different dynamic charac-teristics between the two [12]. It is unknown whetherthe same thing happens with learned facial expressions.An automated system would enable exploration of suchresearch questions.

C. The need for spontaneous facial expression databases

The machine perception community is in critical needof standard video databases to train and evaluate systemsfor automatic recognition of facial expressions. An im-portant lesson learned from speech recognition researchis the need for large, shared databases for training, testing,and evaluation, without which it is extremely difficultto compare different systems and to evaluate progress.Moreover, these databases need to be typical of real worldenvironments in order to train data-driven approaches andfor evaluating robustness of algorithms. An important stepforward was the release of the Cohn-Kanade databaseof FACS coded facial expressions [28], which enableddevelopment and comparison of numerous algorithms.Two more recent databases also make a major contributionto the field: The MMI database which enables greatertemporal analysis as well as profile views [38], as wellas the Lin database which contains 3D range data forprototypical expressions at a variety of intensities [30].However, all of these databases consist of posed facialexpressions. It is essential for the progress of the field tobe able to evaluate systems on databases of spontaneousexpressions. As described above, spontaneous expressionsdiffer from posed expressions in both which musclesare moved, and in the dynamics of those movements.Development of these databases is a priority that requiresjoint effort from the computer vision, machine learning,and psychology communities. A database of spontaneousfacial expressions collected at UT Dallas [34] was a

Figure 2. Overview of fully automated facial action coding system.

significant contribution in this regard. The UT Dallasdatabase elicited facial expressions using film clips, andthere needs to be some concurrent measure of expressioncontent beyond the stimulus category since subjects oftendo not experience the intended emotion and sometimesexperience another one (e.g. disgust or annoyance insteadof humor). FACS coding of this database would beextremely useful for the computer vision community. Wepresent here a database of spontaneous facial expressionsthat has been FACS coded using the Facial Action CodingSystem.

D. System overview

Here we describe progress on a system for fullyautomated facial action coding, and show preliminaryresults when applied to spontaneous expressions. This wasthe first system for fully automated expression coding,presented initially in [3], and it extends a line of researchdeveloped in collaboration with Paul Ekman and TerrySejnowski [11]. It is a user independent fully automaticsystem for real time recognition of facial actions fromthe Facial Action Coding System (FACS). The systemautomatically detects frontal faces in the video streamand codes each frame with respect to 20 Action units. Inprevious work, we conducted empirical investigations ofmachine learning methods applied to the related problemof classifying expressions of basic emotions [31]. Wecompared AdaBoost, support vector machines, and lineardiscriminant analysis, as well as feature selection tech-niques. Best results were obtained by selecting a subsetof Gabor filters using AdaBoost and then training SupportVector Machines on the outputs of the filters selectedby AdaBoost. An overview of the system is shown inFigure 2. Here we apply this system to the problem ofdetecting facial actions in spontaneous expressions.

E. Relation to other work

There have been major advances in the computer visionliterature for facial expression recognition over the past15 years. See [22], [36] for reviews. Much of the earlywork on computer vision applied to facial expressionsfocused on recognizing a few prototypical expressionsof emotion produced on command (e.g., ”smile”). Somesystems describe facial expressions in terms of componentmovements, most notably coding standard developed forMPEG4 which focuses on automatic coding of a set of



facial feature points [10]. While coding standards likeMPEG4 are useful for animating facial avatars, behavioralresearch may require more comprehensive information.For example, MPEG4 does not encode some behaviorallyrelevant movements such as the contraction of the orbic-ularis oculi, which differentiates spontaneous from posedsmiles. It also does not encode changes in surface texturesuch as wrinkles, bulges, and shape changes that arecritical for the definition of action units in the FACSsystem. For example, a characteristic pattern of wrinklesand bulges between the brows, as well as the shape of thebrow (arched vs. flat), are important for distinguishing abasic brow raise (AU 1+2) from a fear brow (AU 1+2+4shown in Figure 1, both of which entail upward movementof the brows, but the brow raise is a common behavior thatemphasizes speech, shows engagement in conversation,or indicates a question, whereas the fear brow occurs insituations of danger and sometimes deception [26].

Several research groups have recognized the impor-tance of automatically coding expressions in terms ofFACS [11], [29], [37], [45], [46]. Our approach differsfrom others in that instead of designing special purposeimage features for each facial action, we employ machinelearning techniques for data-driven facial expression clas-sification. These machine learning algorithms are appliedto image-based representations. Image-based machinelearning methods have been shown to be highly effectivefor machine recognition tasks such as face detection [47],and do not suffer from drift which is a major obstacleto tracking methods. In this paper we show that suchsystems capture information about action unit intensitythat can be employed for analyzing facial expressiondynamics. The image-based representation employed hereis the output of a bank of Gabor filters, although inprevious work we have applied machine learning to theimage features as well, and found that the Gabor featuresare related to those developed by machine learning [2].Learned classifiers taking such representations as inputare sensitive not only to changes in position of facialfeatures, but also to changes in image texture such asthose created by wrinkles, bulges, and changes in featureshapes. We found in practice that image-based representa-tions contain more information for facial expression thanrepresentations based on the relative positions of a finiteset of facial features. For example, our basic emotionrecognizer [31] gave a higher performance on the Cohn-Kanade dataset than an upper-bound on feature trackingcomputed by another group based on manual feature pointlabeling. We distinguish here feature-point tracking fromthe general category of motion-based representations. Onemay describe motion with spatio-temporal Gabor filters,for example, resulting in a representation related to theone presented here. At issue is whether reducing theimage to a finite set of feature positions is a goodrepresentation. Ultimately, combining image-based andmotion based representations may be the most powerful.y

Tian et al. [45] employ traditional computer visiontechniques for state-of-the-art feature tracking. In this

approach, specialized image features such as contourparameters are defined for each desired facial action.Pantic and Rothcrantz [37] use robust facial featuredetection followed by an expert system to infer facialactions from the geometry of the facial features. Morerecent work from this group presented approaches tomeasuring facial actions in profile views, and recognizingexpression dynamics from temporal rules [35]. Theirapproach is more heuristic than the data-driven systempresented here. A strength of data-driven systems is thatthey learn the variations in appearance of a facial actiondue to differences in physiognomy, age, and elasticity,and also when an action occurs in combination with otherfacial actions. Nonlinear support vector machines have theadded advantage of being able to handle multimodal datadistributions which can arise with action combinations,provided that the class of kernel is well matched to theproblem. A group at MIT presented a prototype systemfor fully automated FACS coding that used infrared eyetracking to register face images [29]. The recognitioncomponent is similar in spirit to the one presented here,employing machine learning techniques on image-basedrepresentations. Kapoor et al. used PCA (eigenfeatures)as the feature vector, whereas we previously found thatPCA was a much less effective representation than Gaborwavelets for facial action recognition (see [11], [31]. Morerecently, [46], applied a dynamical Bayesian model tothe output of a front-end FACS recognition system basedon the one developed in our laboratory [3], [4]. While[46] showed that AU recognition benefits from learningcausal relations between AU’s in the training database, theanalysis was developed and tested on a posed expressiondatabase. It will be important to extend such work tospontaneous expressions for the reasons described above.

II. A UTOMATED SYSTEM

A. Real-time Face Detection

We developed a real-time face detection system thatemploys boosting techniques in a generative frame-work [23] and extends work by [47]. Enhancementsto [47] include employing Gentleboost instead of Ad-aBoost, smart feature search, and a novel cascade train-ing procedure, combined in a generative framework.Source code for the face detector is freely availableat http://kolmogorov.sourceforge.net. Accuracy on theCMU-MIT dataset, a standard public data set for bench-marking frontal face detection systems, is 90% detectionsand 1/million false alarms, which is state-of-the-art accu-racy. The CMU test set has unconstrained lighting andbackground. With controlled lighting and background,such as the facial expression data employed here, detec-tion accuracy is much higher. All faces in the trainingdatasets, for example, were successfully detected. Thesystem presently operates at 24 frames/second on a 3 GHzPentium IV for 320x240 images.

The automatically located faces were rescaled to 96x96pixels. The typical distance between the centers of theeyes was roughly 48 pixels. Automatic eye detection



[23] was employed to align the eyes in each image. Theimages were then passed through a bank of Gabor filters8 orientations and 9 spatial frequencies (2:32 pixels percycle at 1/2 octave steps) (See [31]). Output magnitudeswere then passed to the classifiers. No feature selectionwas performed for the results presented here, although itis ongoing work that will be presented in another paper.

B. Facial Action Classification

Facial action classification was assessed for two clas-sifiers: Support vector machines (SVM’s) and AdaBoost.

a) SVM’s.: SVM’s are well suited to this task be-cause the high dimensionality of the Gabor representationO(105) does not affect training time, which dependsonly on the number of training examples O(102). In ourprevious work, linear, polynomial, and radial basis func-tion (RBF) kernels with Laplacian, and Gaussian basisfunctions were explored [31]. Linear and RBF kernelsemploying a unit-width Gaussian performed best on thattask. Linear SVMs are evaluated here on the task of facialaction recognition.

b) AdaBoost.:The features employed for the Ad-aBoost AU classifier were the individual Gabor filters.This gave 9x8x48x48= 165,888 possible features. A sub-set of these features was chosen using AdaBoost. On eachtraining round, the Gabor feature with the best expres-sion classification performance for the current boostingdistribution was chosen. The performance measure wasa weighted sum of errors on a binary classification task,where the weighting distribution (boosting) was updatedat every step to reflect how well each training vector wasclassified. AdaBoost training continued until 200 featureswere selected per action unit classifier. The union of allfeatures selected for each of the 20 action unit detectorsresulted in a total of 4000 features.

III. FACIAL EXPRESSION DATA

A. The RU-FACS Spontaneous Expression Database

Mark Frank, in collaboration with Javier Movellan andMarian Bartlett, has collected a dataset of spontaneousfacial behavior with rigorous FACS coding. The datasetconsists of 100 subjects participating in a ’false opin-ion’ paradigm. In this paradigm, subjects first fill outa questionnaire regarding their opinions about a socialor political issue. Subjects are then asked to either tellthe truth or take the opposite opinion on an issue wherethey rated strong feelings, and convince an interviewerthey are telling the truth. Interviewers were retired policeand FBI agents. A high-stakes paradigm was created bygiving the subjects $50 if they succeeded in fooling theinterviewer, whereas if they were caught they were toldthey would receive no cash, and would have to fill outa long and boring questionnaire. In practice, everyonereceived a minimum of $10 for participating, and no onehad to fill out the questionnaire. This paradigm has beenshown to elicit a wide range of emotional expressions aswell as speech-related facial expressions [26]. This dataset

is particularly challenging both because of speech-relatedmouth movements, and also because of out-of-plane headrotations which tend to be present during discourse.

Subjects faces were digitized by four synchronizedDragonfly cameras from Point Grey. (See Figure 3). Theanalysis in this paper was conducted using the videostream from the frontal view camera. Two minutes of eachsubject’s behavior is being FACS coded by two certifiedFACS coders. FACS codes include the apex frame aswell as the onset and offset frame for each action unit(AU). To date, 33 subjects have been FACS-coded. Herewe present preliminary results for a system trained ontwo large datasets of FACS-coded posed expressions, andtested on the spontaneous expression database. Futurework will include spontaneous expressions in training aswell. Here we explore how well a system trained on posedexpressions under controlled conditions performs whenapplied to real behavior.

Figure 3. Sample synchronized camera views from the RU-FACSspontaneous expression database.

B. Posed expression databases

Because the spontaneous expression database did notyet contain sufficient labeled examples to train a data-driven system, we trained the system on a larger set oflabeled examples from two FACS-coded datasets of posedimages. The first dataset was Cohn and Kanade’s DFAT-504 dataset [28]. This dataset consists of 100 universitystudents ranging in age from 18 to 30 years. 65% werefemale, 15% were African-American, and 3% were Asianor Latino. Videos were recoded in analog S-video usinga camera located directly in front of the subject. Subjectswere instructed by an experimenter to perform a seriesof 23 facial displays. Subjects began each display with aneutral face. Before performing each display, an experi-menter described and modeled the desired display. Imagesequences from neutral to target display were digitizedinto 640 by 480 pixel arrays with 8-bit precision forgrayscale values. The facial expressions in this datasetwere FACS coded by two certified FACS coders.



The second dataset consisted of directed facial actionsfrom 24 subjects collected by Ekman and Hager. (See[11].) Subjects were instructed by a FACS expert on thedisplay of individual facial actions and action combina-tions, and they practiced with a mirror. The resulting videowas verified for AU content by two certified FACS coders.

IV. T RAINING

The combined dataset contained 2568 training exam-ples from 119 subjects. Separate binary classifiers, one foreach AU, were trained to detect the presence of the AUregardless of the co-occurring AU’s. We refer to this ascontext-independent recognition. Positive examples con-sisted of the last frame of each sequence which containedthe expression apex. Negative examples consisted of allapex frames that did not contain the target AU plus neutralimages obtained from the first frame of each sequence, fora total of 2568-N negative examples for each AU.

V. GENERALIZATION PERFORMANCEWithin DATASET

We first report performance for generalization tonovel subjectswithin the Cohn-Kanade and Ekman-Hagerdatabases. Generalization to new subjects was tested usingleave-one-subject-out cross-validation in which all imagesof the test subject were excluded from training. Resultsfor the AdaBoost classifier are shown in Table I. Systemoutputs were the output of the AdaBoost discriminantfunction for each AU. All system outputs above thresholdwere treated as detections.

Figure 4. ROC curves for 8 AU detectors, tested on posed expressions.

The system obtained a mean of 91% agreement withhuman FACS labels. Overall percent correct can be anunreliable measure of performance, however, since itdepends on the proportion of targets to non-targets, andalso on the decision threshold. In this test, there was a fargreater number of non-targets than targets, since targetswere images containing the desired AU (N in Table I, andnon-targets were all images not containing the desired AU(2568-N). A more reliable performance measure is areaunder the ROC (receiver-operator characteristic curve.)This curve is obtained by plotting hit rate (true positives)against false alarm rate (false positives) as the decisionthreshold varies. See Figure 4. The area under this curveis denoted A′. A′ is equivalent to percent correct in a 2-alternative forced choice task, in which the system mustchoose which of two options contains the target on eachtrial. Mean A′ for the posed expressions was 92.6.

TABLE I.PERFORMANCE FOR POSED EXPRESSIONS.

Shown is fully automatic recognition of 20 facial actions, generalizationto novel subjects in the Cohn-Kanade and Ekman-Hager databases. N:Total number of positive examples. P: Percent agreement with HumanFACS codes (positive and negative examples classed correctly). Hit, FA:Hit and false alarm rates. A′: Area under the ROC. The classifier wasAdaBoost.

AU Name N P Hit FA A′

1 Inn. brow raise 409 92 86 7 952 Out. brow raise 315 88 85 12 924 Brow lower 412 89 76 9 915 Upper lid raise 286 92 88 7 966 Cheek raise 278 93 86 6 967 Lower lid tight 403 88 89 12 959 Nose wrinkle 68 100 88 0 10010 Lip Raise 50 97 29 2 9011 Nasolabial 39 94 33 4 7412 Lip crnr. pull 196 95 93 5 9814 Dimpler 32 99 20 0 8515 Lip crnr. depr. 100 85 85 14 9116 Lower Lip depr. 47 98 29 1 9217 Chin raise 203 89 86 10 9320 Lip stretch 99 92 57 6 8423 Lip tighten 57 91 42 8 7024 Lip press 49 92 64 7 8825 Lips part 376 89 83 9 9326 Jaw drop 86 93 58 5 8527 Mouth stretch 81 99 100 1 100

Mean 90.9 80.1 8.2 92.6

Figure 5. System performance (area under the ROC) for posed facialactions. Actions were sorted in order of detection performance. High,middle, and low-performing AU’s are illustrated.

Figure 5 contains a graphical depiction of performancesorted in order of A’. High, middle, and low performingaction units are illustrated. We note that both the highestand lowest performance was obtained with lower-faceAU’s (AU’s 9-27), while mid-range performance wasobtained for brow and eye region actions (AU’s 1-7).

A. Effect of training set size

We next investigated to what degree the performancevariation related to training set size. Inspection of theROC curves in Figure 4 suggests a dependence of sys-



tem performance on the number of training examples.Figure /reffig:numex plots area under the roc againsttraining set size for all 20 AU’s tested. The scatter plot iselbow-shaped, where action units with the fewest trainingexamples also had the lowest performance, and then theplot flattens for training set sizes substantially greater than100. Action units 27 and 9 are reliably detected despiterelatively small training set sizes, suggesting these twomay be easier to detect.

Figure 6. Scatter plot of system performance (area under theROC)against training set size for the 20 AU’s in Table I.

VI. GENERALIZATION TO SPONTANEOUS

EXPRESSIONS

We then tested the system described in Section IV ona new dataset of spontaneous expressions, the RU-FACSdataset. The dataset included speech related mouth andface movements, and significant amounts of in-plane andin-depth rotations.

A. characterization of head pose during spontaneousexpression

Figure 7. Distribution of head poses in frontal view camera duringexpression apex.

TABLE II.HEAD POSE DISTRIBUTION DURING EXPRESSION APEX.

Mean sd 67% Interval 96% Interval

Yaw -6.50 8.90 [-150, 20] [-240, 110]Pitch -8.50 7.10 [-160, 20] [-230, 50]Roll -2.80 7.40 [-100, 40] [-180, 20]

In contrast to the highly controlled conditions of posedexpression databases, in spontaneous behavior the headpose of subjects varies from frontal. In order to character-ize the distribution of head pose during spontaneous facialexpressions, we manually labeled the head pose for theaction unit apex frames for 473 images from 21 subjects.Head pose was labeled by rotating a 3D head model withthe arrow keys until it matched the head pose in theimage. To assist alignment, an initial estimate of headpose was calculated from eye, nose, and mouth positions.In addition, the face image was projected onto the 3Dhead model and then back into the image plane from theestimated pose, in order to match the projected face withthe face in the image.

Figure 7 shows the distribution of yaw, pitch, and rollduring expression apex. The histograms show that eachof these three head orientation measures ranges fromapproximately−300 to 200. Table II gives the mean andstandard deviations of yaw, pitch, and roll. The mean headpose in this dataset is80 down and60 to the left, whichis likely a result of the constraints of camera placement,in which the camera was placed over the right shoulderof the interviewer. Roll (or in-plane rotation) is the moststraightforward to correct in computer vision systems. Inboth yaw and pitch, approximately 30% of all apex frameswere between150 and 250 from frontal, and no imagesin our sample were rotated beyond300

B. AU recognition: Spontaneous Expressions

Preliminary recognition results are presented for 12subjects. This data contained a total of 1689 labeledevents, consisting of 33 distinct action units, 19 of whichwere AU’s for which we had trained classifiers. Facedetections were accepted if the face box was greater than150 pixels width, both eyes were detected with positiveposition, and the distance between the eyes was> 40pixels. This resulted in faces found for 95% of the videoframes. Most non-detects occurred when there was headrotations beyond±100 or partial occlusion. All detectedfaces were passed to the AU recognition system.

Here we present benchmark performance of the basicframe-by-frame system on the video data. Figure 8 showssample system outputs for one subject, and performanceis shown in Figure 5 and Table III. Performance wasassessed several ways. First, we assessed overall percentcorrect for each action unit on a frame-by-frame basis,where system outputs that were above threshold inside theonset and offset interval indicated by the human FACScodes, and below threshold outside that interval were



a

b

Figure 8. Sample system outputs for a 10-second segment containinga brow-raise (FACS code 1+2). System output is shown for AU 1 (left)and AU 2 (right). Human codes are overlayed for comparison (onset,apex, offset).

TABLE III.RECOGNITION OF SPONTANEOUS FACIAL ACTIONS.

AU: Action unit number. N: Total number of testing examples.P: Percentcorrect over all frames. Hit, FA: Hit and false alarm rates. A′: Area underthe ROC. A′

∆: Area under the ROC for interval analysis (see text). The

classifier was AdaBoost.

AU N P Hit FA A′ A′

∆

1 169 87 35 9 78 832 153 84 29 13 62 684 32 97 15 2 74 845 36 97 7 1 71 766 50 92 32 4 90 927 46 91 12 7 64 669 2 99 0 0 88 9310 38 95 0 0 62 6511 3 99 0 0 73 8312 119 86 45 7 86 8814 87 94 0 0 70 7715 77 94 23 4 69 7316 5 99 0 0 63 5717 121 93 15 2 74 7620 12 99 0 0 66 6923 24 98 0 0 69 7524 68 95 7 3 64 6325 200 54 68 50 70 7326 144 91 2 1 63 64

Mean 93 15 5 71 75

considered correct. This gave an overall accuracy of 93%correct across AU’s for the AdaBoost classifier. Mean areaunder the ROC was .71.

Next an interval analysis was performed, which wasintended to serve as a baseline for future analysis of out-

Figure 9. AU recognition performance (area under the ROC) forspontaneous facial actions. Performance is overlayed on the posed resultsof Figure 5.

put dynamics. The interval analysis measured detectionson intervals of length I. Here we present performance forintervals of length 21 (10 on either side of the apex), butperformance was stable for a range of choices of I. Atarget AU was treated as present if at least 6/21 frameswere above threshold, where the threshold was set to 1standard deviation above the mean. Negative examplesconsisted of the remaining 2 minute video stream for eachsubject, outside the FACS coded onset and offset intervalsfor the target AU, parsed into intervals of 21 frames. Thissimple interval analysis raised the area under the ROC to.75.

TABLE IV.COMPARISON OFADABOOST TO LINEARSVM’ S.

The task is AU classification in the spontaneous expression database.A′

∆: Area under the ROC for interval analysis (see text).

AdaBoost SVMAU N A′ A′

∆A′ A′

∆

1 169 78 83 73 832 153 62 68 63 764 32 74 84 74 865 36 71 76 63 7310 38 62 65 60 7112 119 86 88 84 9014 87 70 77 65 7320 12 66 69 60 74

Mean 71.1 76.3 67.8 78.3

C. AdaBoost v. SVM performance

Table IV compares AU recognition performance withAdaBoost to a linear SVM. In previous work with posedexpressions of basic emotions, AdaBoost performed sim-ilarly to SVM’s, conferring a marginal advantage overthe linear SVM [31]. Here we support this finding forrecognition of action units in spontaneous expressions.AdaBoost had a small advantage over the linear SVMwhich was statistically significant on a paired t-test (



t(7)=3.1, p=.018). A substantial performance increase wasincurred for both classifiers by employing the intervalanalysis. Here the outputy was first converted to Z-scores for each subjectz = (y − µ/σ), and then z wasintegrated over a window of 11 frames. The The temporalinformation in the classifier outputs contain considerableinformation that we intend to exploit in future work.

VII. T HE MARGIN PREDICTS EXPRESSION INTENSITY

Figure 10 shows a sample of system outputs for a 2minute 20 second continuous video stream from the spon-taneous expression database. Inspection of such outputstreams suggested that the system output, which was thedistance to the separating hyperplane (the margin), con-tained information about expression intensity. A strongerrelationship was observed for the outputs of the poseddata, which had less noisy image conditions. Systemoutputs for full image sequences of test subjects fromthe posed data are shown in Figure 11. Although eachindividual image is separately processed and classified,the outputs change smoothly as a function of expressionmagnitude in the successive frames of each sequence.

Figure 10. Output trajectory for a 2 minute 20 sec. video (6000 frames),for one subject and one action unit. Shown is the margin (the distanceto the separating hyperplane). The human FACS labels are overlaid forcomparison. Blue stars indicate the frame at which the AU apex wascoded. The frames within the onset and offset of the AU are shown inred. Letters A-E indicate AU intensity, with E highest.

a

b

Figure 11. Automated FACS measurements for full image sequences.a. Surprise expression sequences from 4 subjects containing AU’s 1,2and 5. b. Disgust expression sequences from 4 subjects containing AU’s4,7 and 9.

Intensity correlation: Posed data:A correlation anal-ysis was performed in order to explicitly measure therelationship between the output margin and expressionintensity. In order to assess this relationship in low noiseconditions, we first performed the analysis for posed ex-pressions. The posed data contains no speech, negligiblehead rotation, and consequently less luminance variationthan the spontaneous data. In addition, the posed databasewas the training database, which enabled us to measurethe degree to which the SVM learned about expressionintensity for validation subjects in the same database asthe training set.

Ground truth for action unit intensity was measuredas follows: Five certified FACS coders labeled the actionintensity for 108 images from the Ekman-Hager database.The images were four upper-face actions (1, 2, 4, 5) andtwo lower-face actions (10, 20), displayed by 6 subjects.Three images from each sequence were displayed: oneimmediately after onset, one at apex, and one intermediateframe. Images were presented in random order, and theFACS experts were asked to label both the AU and the AUintensity. In keeping with FACS coding procedures, theexperts scored intensity on an A through E scale, whereA is lowest, and E is highest. The experts did not alwaysagree with the FACS label in the database, particularlyfor the lowest intensity frames. Disagreement rate was5.3% for the upper-face actions and 8.5% for the lower-face actions. Intensities were included in the subsequentanalysis only for frames on which the experts agreed withthe AU label in the database.

We first measured the degree to which expert FACScoders agree with each other on intensity. Correlationswere computed separately for each action unit. Correla-tions were computed between intensity scores by eachpair of experts, and the mean correlation was computedacross all expert pairs. The results are given in Table V.Because individual faces differ in the amount of wrinklingand movement for a given facial action, the correlationswere computed two ways: within-subject and between-subject, where ’subject’ refers to the person displaying theaction unit. Within-subject correlations effectively removedifferences in mean and variance between individuals, andis similar to computing a Z-score prior to correlating. Forthe within-subject analysis, correlations were computedseparately for each subject, and the mean was computedacross subjects. Mean correlation between expert FACScoders within subject was 0.84.

Correlations of the automated system with the humanexpert intensity scores were next computed. The SVM’swere retrained on the even-numbered subjects of theCohn-Kanade and Ekman-Hager datasets, and then testedon the odd-numbered subjects of the Ekman-Hager set,and vice versa. Correlations were computed between theSVM margin and the intensity ratings of each of the fiveexpert coders. The analysis was again performedwithinsubject, and then means were computed for each AUby collapsing across subject. The results are shown inTable VA. Overall mean correlation between the SVM



margin and the expert FACS coders was 0.83, which wasvery similar to the human-human correlation of .84.

The analysis was next repeated between-subjects, andthe results are shown in Table VB. For the between-subject analysis, all subjects displaying the action unitwere included in a single correlation for each AU. We seethat the expert agreement dropped about 10% to .73 bydoing the between-subject correlation. For the SVM, theagreement with expert humans dropped to .53. This showsthat the system will benefit from online learning of scaleand threshold for each subject, since the within-subjectanalysis effectively removed mean and scale differencesbetween subjects.

TABLE V.HUMAN -SVM INTENSITY CORRELATIONS.

Expert-Expert is the mean correlation (r) across 5 human FACS experts.SVM-Expert is the mean correlation of the margin with intensity ratingfrom each of the 5 experts. A. Correlations were computedwithinsubject displaying the facial action. B. Correlations werecomputedacrosssubjects displaying the facial action.

A. Within Subject, posedAction Unit

1 2 4 5 10 20 Mean

Expert-Expert .92 .77 .85 .72 .88 .88 .84SVM-Expert .90 .80 .84 .86 .79 .79 .83

B. Between Subject, posedAction Unit

1 2 4 5 10 20 Mean

Expert-Expert .77 .64 .79 .63 .75 .79 .73SVM-Expert .47 .85 .45 .39 .36 .69 .53

TABLE VI.HUMAN -SVM INTENSITY CORRELATIONS, SPONTANEOUS DATA.

Action Unit1 2 4 5 10 12 14 20 Mean

.31 .09 .38 .51 .29 .75 .16 .33 .35

Intensity correlation: Spontaneous data:The cor-relation analysis was then repeated for the spontaneousexpression data. This effectively tests how well the rela-tionship between the margin and the intensity generalizesto a new dataset, and also how well it holds up in thepresence of noise from speech and head movements. Theintensity codes in the RU-FACS dataset were used asground truth for action intensity. Correlations were com-puted between the margin of the linear SVM and the AUintensity as coded by the human coders for each subjectfor the 8 AU’s shown in Table IV. As above, correlationswere computed within subject, and then collapsed acrosssubject to provide a mean correlation for each AU. Theoverall mean correlation was r=0.35. There was muchvariability in the correlations across AU. AU 12, forexample, had a correlation of 0.75 between the margin

and FACS intensity score. The correlations for the spon-taneous expression data were overall substantially smallerthan for the posed data. Nevertheless, for some facialactions the system extracted a signal about expressionintensity on this very challenging dataset.

VIII. P ERFORMANCE FACTORS

A. Effect of Compression

For many applications of automatic facial expressionanalysis, image compression is desirable in order to makean inexpensive, flexible system. The image analysis meth-ods employed in this system, such as Gabor filters, maybe more robust to lossy compression compared to otherimage analysis methods such as optic flow. We there-fore investigated the relationship between AU recognitionperformance and image compression. Detectors for threeaction units (AU 1, AU2, and AU4) were compared whentested at five levels of compression: No loss (original bmpimages), and 4 levels of jpeg compression quality: 100%,75in Figure 12. Performance remained consistent acrosssubstantial quantities of lossy compression. This findingis of practical importance for system design.

Figure 12. Effects of compression on AU recognition performance.

B. Handling AU combinations

When they occur together, facial actions can take ona very different appearance from when the occur inisolation. A similar effect happens in speech recognitionwith phonemes, and is called a co-articulation effect. TheSVM’s in this study were trained to detect an action unitregardless of whether it occurs alone or in combinationwith other action units. However, as in speech recognition,we may obtain better performance by training dedicateddetectors for certain AU combinations.

An AU combination analysis was performed on thethree brow action units (1, 2, 4). The analysis wasperformed on a set of linear SVM’s trained on half ofthe Ekman-Hager database and tested on the other half.AU’s 1 and 2 pull the brow up (centrally and laterally,respectively), whereas AU 4 pulls the brows together anddown using primarily the corrugator muscle at the bridgeof the nose. The appearance of AU 4 changes dramaticallydepending on whether it occurs alone or in combinationwith AU 1 and 2. Inspection of Table VII shows thatperformance may benefit from treating AU4 separately.We also see that it is not necessary or desirable to treat all



AU combinations separately. For example, performancedoes not benefit from treating AU 1 and AU 2 separately.

TABLE VII.COMBINATION ANALYSIS

Performance for a linear SVM trained to detect specific combinationsof the brow AU’s. Target = target set for training. Non-targets duringtraining were all other AU’s in the Ekman-Hager database. There wereno spontaneous examples of 2+4.

Posed Spont.Target A’ A’

AU 1 alone or in comb. 93.5 72.6Individual AU 1 only 72.7 60.4



1+2 98.2 70.51+4 89.4 60.12+4 95.8 –

1+2+4 90.8 68.5

IX. CONCLUSIONS

The current state of the art in automatic face andgesture recognition suggests that user independent, fullyautomatic real time coding of facial expressions in thecontinuous video stream is an achievable goal withpresent computer power. The system presented here oper-ates in real time. Face detection runs at 24 frames/secondin 320x240 images on a 3 GHz Pentium IV. The AUrecognition step operates in less than 10 msec per actionunit.

The next step in the development of automatic facialexpression recognition systems is to begin to apply themto spontaneous expressions for real applications. Spon-taneous facial behavior differs from posed expressionsboth in which muscles are moved, and in the dynamicsof those movements. These differences are described inmore detail in the Introduction. The step to spontaneousbehavior involves handling variability in head pose, andoften the presence of speech, in addition to handling thewide variety of facial muscle constellations that occur innatural behavior.

This paper presented preliminary results for a fullyautomated facial action detection system on a databaseof spontaneous facial expressions. These results providea benchmark for future work on spontaneous expressionvideo. The system was able to extract information aboutfacial actions in this dataset despite substantive differ-ences between the spontaneous expressions and the poseddata on which it was trained. While at this time thereis insufficient FACS-labeled spontaneous expressions to

support training of data-driven systems exclusively onspontaneous expressions, a combined training approach ispossible, and an important next step is to explore systemstrained on a combined posed and spontaneous dataset.Preliminary results in our lab using such a combineddataset, and testing with cross-validation, show that per-formance is substantively improved on the spontaneousexpression data, while performance declines on the poseddata. This finding reinforces the differences betweenposed and spontaneous expressions.

A significant finding from this paper is that data-driven classifiers such as SVM’s learned informationabout expression intensity. The distance to the separatinghyperplane, the margin, was significantly correlated withfacial action intensity codes. Current work in our labis showing a similar relationship with Adaboost, wherein the case of AdaBoost, it is the likelihood ratios inthe Adaboost discriminant function that correlate withmeasures of expression intensity. The system thereforeis able to provide information about facial expression dy-namics in the frame-by-frame intensity information. Thisinformation can be exploited for deciding the presence ofa facial action and decoding the onset, apex, and offset.It will also enable explorations of the dynamics of facialbehavior, as discussed below.

The accuracy of automated facial expression mea-surement in spontaneous behavior may be considerablyimproved by 3D alignment of faces. Moreover, infor-mation about head movement dynamics is an importantcomponent of nonverbal behavior, and is measured inFACS. Members of this group have developed techniquesfor automatically estimating 3D head pose in a generativemodel and for aligning face images in 3D [32]. We arealso exploring feature selection techniques. Our previouswork with expressions of basic emotion showed thatfeature selection by AdaBoost significantly enhanced bothspeed and accuracy of SVM’s [31]. We are presently ex-ploring whether such advantages found for basic emotionrecognition carry over to the task of AU detection inspontaneous expressions.

The system presented here is fully automated, and per-formance rates for posed expressions compare favorablywith other systems tested on the Cohn-Kanade dataset thatemployed varying levels of manual registration. A stan-dard database of FACS coded spontaneous expressionswould be of great benefit to the field and we are preparingto make the RU-FACS spontaneous expression databaseavailable to the research community.

A. Applications

Man-machine interaction for education:A majorthrust of research in human-computer and human-robotinteraction is the development of tools for education.Expression measurement tools enable automated tutoringsystems that recognize the emotional and cognitive stateof the pupil and respond accordingly. Such systems wouldalso assist robots and animated agents to establish socialresonance. Research has shown that behaviors including



mirroring facial expressions and head movements assistsin generating social rapport and can lead to increasedinformation transfer between humans (e.g. [5], [7]. Suchbehaviors may increase the effectiveness of animatedagents and robots designed for education environments.In addition, there is evidence suggesting that automatictutors can become more effective if they use informationabout the eye movements of the students [1], [43].

Behavioral Science and Psychiatry:Tools for auto-matic expression measurement would enable tremendousnew research activity not only in emotion, but also socialpsychology, development, cognitive neuroscience, psy-chiatry, education, human-machine communication, andhuman social dynamics. These tools will bring aboutparadigmatic shifts in these fields by making facial ex-pression more accessible as a behavioral measure. Newresearch activities enabled by this technology includestudying the cognitive neuroscience of emotion, moodregulation, and social interaction; measuring the efficacyof psychiatric treatment including new medications, andstudying facial behavior in psychiatric and developmentaldisorders.

Dynamics of facial behavior:Automated expressionmeasurement tools developed in projects such as theone presented here will enable investigations into thedynamics of human facial expression that were previ-ously infeasible with manual coding. This would allowresearchers to directly address a number of questionskey to understanding the nature of the human emotionaland expressive systems, and their roles interpersonalinteraction, psychopathology, and development. Previousresearch with manual coding has shown differences in thedynamics of spontaneous expressions compared to posed,as well as differences in the facial dynamics of patientswith neuropathology (e.g. schizophrenia). Research hasalso shown that subtle movement differences between feltand unfelt expressions can be a critical indicator of socialfunctioning, of the progress and remission of depression,and of suicide potential, as well as provide signs ofdeception [21], [26]. There are very few experiments ofthis nature because of the time burden of manual codingof dynamics, and the coding that has been done measuresonly coarse information about dynamics.

Security: Automatic facial action measurement hasprofound consequences on law enforcement and counterterrorism. Careful laboratory studies show that many ofthe clues to concealed emotion and deceit currently usedin law enforcement training programs may be quite unre-liable. Moreover, research based on facial action codingshowed that more reliable cues exist in facial behavior[26]. Extracting this information requires detailed analysisof facial expression. Real-time automated coding can non-obtrusively supplement the other information availableto interviewers, screeners, and law enforcement agentsby identifying subtle or conflicted expressions that maybetray someone’s true emotional state. Automatic Expres-sion coding will also enable more thorough investigationof the role of facial expression in deception.

Acknowledgments

Support for this work was provided by NSF IIS-0220141, NSF CNS-0454233, and NRL/HSARPA Ad-vanced Information Technology 55-05-03 to Bartlett,Movellan, and Littlewort. Support for M.G. Frank wasprovided by NSF 0220230 and NSF 0454183. This mate-rial is based upon work supported by the National ScienceFoundation under a Cooperative Agreement/Grant. Anyopinions, findings, and conclusions or recommendationsexpressed in this material are those of the author(s) and donot necessarily reflect the views of the National ScienceFoundation.

REFERENCES

[1] J. R. Anderson. Spanning seven orders of magnitude:a challenge for cognitive modeling.Cognitive Science,26:85–112, 2002.

[2] M.S. Bartlett, G.L. Donato, J.R. Movellan, J.C. Hager,P. Ekman, and T.J. Sejnowski. Image representations forfacial expression coding. In S.A. Solla, T.K. Leen, andK.-R. Muller, editors, Advances in Neural InformationProcessing Systems, volume 12. MIT Press, 2000.

[3] M.S. Bartlett, G. Littlewort, I. Fasel, and J.R. Movellan.Real time face detection and expression recognition: De-velopment and application to human-computer interaction.In CVPR Workshop on Computer Vision and PatternRecognition for Human-Computer Interaction, 2003.

[4] M.S. Bartlett, G. Littlewort, M.G. Frank, C. Lainscsek,I. Fasel, and J. Movellan. Recognizing facial expression:Machine learning and application to spontaneous behavior.In IEEE International Conference on Computer Vision andPattern Recognition, pages 568–573, 2005.

[5] F. Bernieri, J. Davis, R. Rosenthal, and C. Knee. Inter-actional synchrony and rapport: Measuring synchrony indisplays devoid of sound and facial affect.Personalityand Social Psychology Bulletin, 20:303–311, 1994.

[6] A. Brodal. Neurological anatomy: In relation to clinicalmedicine. Oxford University Press, New York, 1981.

[7] T.L. Chartrand and J. A. Bargh. The chameleon effect: theperception-behavior link and social interaction.Journal ofPersonality and Social Psychology, 76:893–911, 1999.

[8] K.D. Craig, S.A. Hyde, and C.J. Patrick. Genuine, sup-pressed, and faked facial behavior during exacerbation ofchronic low back pain.Pain, 46:161–172, 1991.

[9] C. Darwin. The expression of the emotions in man andanimals. Oxford University Press, New York, 1872/1998.3rd Edition, w/ commentaries by Paul Ekman.

[10] P. Doenges, F. Lavagetto, J. Ostermann, I.S. Pandzic, andE. Petajan. Mpeg-4: Audio/video and synthetic graph-ics/audio for real-time, interactive media delivery.ImageCommunications Journal, 5(4), 1997.

[11] G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Se-jnowski. Classifying facial actions.IEEE Transactionson Pattern Analysis and Machine Intelligence, 21(10):974–989, 1999.

[12] Torres E. and Andersen R. Space-time separation duringobstacle-avoidance learning in monkeys. submitted.

[13] P. Ekman. The argument and evidence about universalsin facial expressions of emotion. In D.C. Raskin, editor,Psychological methods in criminal investigation and evi-dence, pages 297–332. Springer Publishing Co, Inc., NewYork, 1989.

[14] P. Ekman.Telling Lies: Clues to Deceit in the Marketplace,Politics, and Marriage. W.W. Norton, New York, 2ndedition, 1991.



[15] P. Ekman. Facial expression of emotion.AmericanPsychologist, 48:384–392, 1993.

[16] P. Ekman, R. J. Davidson, and W. V. Friesen. Theduchenne smile: Emotional expression and brain physi-ology ii. Journal of Personality and Social Psychology,58:342–353, 1990.

[17] P. Ekman and W. Friesen.Facial Action Coding System:A Technique for the Measurement of Facial Movement.Consulting Psychologists Press, Palo Alto, CA, 1978.

[18] P. Ekman and W. V. Friesen. Felt, false, and miserablesmiles. Journal of Nonverbal Behavior, 6:238–252, 1982.

[19] P. Ekman, R.W. Levenson, and W.V. Friesen. Autonomicnervous system activity distinguishes between emotions.Science, 221:1208–1210, 1983.

[20] P. Ekman and E.L. Rosenberg.What the Face Reveals:Basic and Applied Studies of Spontaneous Expressionusing the Facial Action Coding System (FACS). OxfordUniversity Press, New York, 1997.

[21] P. Ekman and E.L. Rosenberg.What the Face Reveals:Basic and Applied Studies of Spontaneous Expression us-ing the Facial Action Coding System (FACS). 2nd Edition.Oxford University Press, New York, 2005.

[22] B. Fasel and J. Luettin. Automatic facial expressionanalysis: A survey. Pattern Recognition, 36:259–275,2003.

[23] Ian R Fasel, Bret Fortenberry, and Javier R Movellan. Agenerative framework for real-time object detection andclassification.Computer Vision and Image Understanding,98, 2005.

[24] M. G. Frank and P. Ekman. Not all smiles are cre-ated equal: The differences between enjoyment and othersmiles.Humor: The International Journal for Research inHumor, 6:9–26, 1993.

[25] M.G. Frank. International Encyclopedia of the Social andBehavioral Sciences, chapter Facial expressions. Elsevier,Oxford, 2002.

[26] M.G. Frank and P. Ekman. Appearing truthful generalizesacross different deception situations.Journal of personalityand social psychology, 86:486–495, 2004.

[27] M. Heller and V. Haynal. The faces of suicidal depres-sion (translation). les visages de la depression de suicide.Kahiers Psychiatriques Genevois (Medecine et HygieneEditors), 16:107–117, 1994.

[28] T. Kanade, J.F. Cohn, and Y. Tian. Comprehensivedatabase for facial expression analysis. InProceedingsof the fourth IEEE International conference on automaticface and gesture recognition (FG’00), pages 46–53, Greno-ble, France, 2000.

[29] Qi Y. Kapoor, A. and R.W. Picard. Fully automatic upperfacial action recognition. InIEEE International Workshopon Analysis and Modeling of Faces and Gestures, 2003.

[30] Y. Lin, X. Wei, Y. Sun, J. Wang, and M. Rosato. A 3dfacial expression database for facial behavior research. InProceedings, International Conference Automatic on Faceand Gesture Recognition, 2006.

[31] G. Littlewort, M.S. Bartlett, I. Fasel, J. Susskind, andJ.R. Movellan. An automatic system for measuring facialexpression in video.Image and Vision Computing, in press.

[32] T. K. Marks, J. Hershey, J. Cooper Roddey, and J. R.Movellan. 3d tracking of morphable objects using con-ditionally gaussian nonlinear filters. InCVPR Workshopon Generative-Model Based Vision, 2004.

[33] A. Meihlke. Surgery of the facial nerve. Saunders,Philadelphia, 1973.

[34] AJ O’Toole, J Harms, SL Snow, DR Hurst, MR Pappas JHAyyad, and H Abdi. A video database of moving facesand people. IEEE Transactions on Pattern Analysis andMachine, 27(5):812–816, 2005.

[35] M. Pantic and I. Patras. Dynamics of facial expression:Recognition of facial actions and their temporal segments

from face profile image sequences.IEEE Transactionson Systems, Man and Cybernetics-Part B, 36(2):433–449,2006.

[36] M. Pantic and J.M. Rothkrantz. Automatic analysis offacial expressions: State of the art.IEEE Transactions onPattern Analysis and Machine Intelligence, 22(12):1424–1445, 2000.

[37] M. Pantic and J.M. Rothkrantz. Facial action recognitionfor facial expression analysis from static face images.IEEE Transactions on Systems, Man and Cybernetics,34(3):1449–1461, 2004.

[38] M. Pantic, M.F. Valstar, R. Rademaker, and L. Maat. ’web-based database for facial expression analysis. InProc.IEEE Int’l Conf. on Multimedia and Expo (ICME ’05),pages 317–321, 2005.

[39] R.W. Picard.Affective Computing. MIT Press, 1997.[40] W. E. Rinn. The neuropsychology of facial expression: A

review of the neurological and psychological mechanismsfor producing facial expressions.Psychological Bulletin,95(1):52–77, 1984.

[41] Morecraft RJ, Louie JL, Herrick JL, and Stilwell-MorecraftKS. Cortical innervation of the facial nucleus in the non-human primate. a new interpretation of the effects of strokeand related subtotal brain trauma on the muscles of facialexpression.Brain, 124:176–208, 2001.

[42] E.L. Rosenberg, P. Ekman, W. Jiang, M. Babyak, et al.Linkages between facial expressions of anger and transientmyocardial ischemia in men with coronary artery disease.American Psychological Assn, US. Emotion, 1(2):107–115,2001.

[43] D. D. Salvucci and J. R. Anderson. Intelligent gaze-addedinterfaces. in human factors in computing systems. InCHIConference Proceedings, pages 273–280, New York, 2000.ACM Press.

[44] M. A. Sayette, D. W. Smith, M.J. Breiner, and G. T Wilson.The effect of alcohol on emotional response to a socialstressor.Journal of Studies on Alcohol, 53:541–545, 1992.

[45] Y.L. Tian, T. Kanade, and J.F. Cohn. Recognizing actionunits for facial expression analysis.IEEE Transactionson Pattern Analysis and Machine Intelligence, 23:97–116,2001.

[46] Y. Tong, W. Liao, and Q. Ji. Inferring facial action unitswith causal relations. InProceedings, Computer Visionand Pattern Recognition, 2006.

[47] P. Viola and M. Jones. Robust real-time object detection.In ICCV Second International Workshop no Statistical andConceptual Theories of Vision., 2001.



Marian Stewart Bartlett is As-sociate Research Professor atthe Institute for Neural Com-putation, UCSD, where sheco-directs the Machine Per-ception Lab. She studies learn-ing in vision, with applica-tion to face recognition andexpression analysis. She hasauthored over 30 articles inscientific journals and refer-eed conference proceedings, aswell as a book, Face Im-

age Analysis by Unsupervised Learning, published byKluwer in 2001. Dr. Bartlett obtained her Bachelor’sdegree in Mathematics in 1988 from Middlebury College,and her Ph.D. in Cognitive Science and Psychologyfrom University of California, San Diego, in 1998. Herthesis work was conducted with Terry Sejnowski atthe Salk Institute. She has also published papers in vi-sual psychophysics with Jeremy Wolfe, neuropsychologywith Jordan Grafman, perceptual plasticity with V.S.Ramachandran, machine learning with Javier Movellan,automatic recognition of facial expression with PaulEkman, cognitive models of face perception with JimTanaka, and the visuo-spatial properties of faces andAmerican Sign Language with Karen Dobkins.

Gwen C. Littlewort holds anN.S.F. Advance fellowship atthe Machine Perception Labo-ratory, U.C.S.D., where she in-vestigates automatic facial ex-pression recognition using ma-chine learning techniques. Shehas a BSc in Electrical Engi-neering from U. Capetown anda PhD in Physics from Oxford.

Mark Frank is Associate Pro-fessor, Communication De-partment, University at Buf-falo State University of NewYork. His research interests in-clude nonverbal behavior incommunication, with an em-phasis on facial expression. Hereceived a B.A. in Psychologyfrom SUNY Buffalo in 1983,and his Ph.D. in Social Psy-chology from Cornell Univer-sity in 1989. He completed apostdoc with Paul Ekman at

the University of California, San Francisco Departmentof Psychiatry in 1992. Dr. Frank has studied nonverbalbehavior in deception for many years, and has worked ex-tensively with law enforcement groups, including helpingthem develop their interviewing and training programsin the context of counter-terrorism. He has also givenworkshops to courts nationally and internationally.

Claudia Lainscsek received theM.S. and PhD degrees inPhysics and Technical Sci-ences from the University ofTechnology in Graz, Austria in1992 and 1999, respectively.She has been at the Insti-tute for Neural Computationsince 2004, working on Non-linear Dynamical Systems the-ory and Facial Action recogni-

tion.Ian Fasel is a postdoctoral re-searcher at the Institute for NeuralComputation, University of Cali-fornia, San Diego. His primary re-search interests are in learning andvision, and in particular what itsrole is in human social interac-tion and development. He receivedhis B.S. in Electrical Engineeringat University of Texas, Austin, in1999, and his Ph.D. in Cognitive

Science at the University of California, San Diego in1996.

Javier R. Movellan is FullProject Scientist at the In-stitute for Neural Compu-tation and Director of theMachine Perception Labora-tory at the University ofCalifornia San Diego. Previ-ous to this he was a Ful-bright fellow at UC Berke-ley (1984-1989), a Carnegie-Mellon University research as-

sociate (1989-1993) and an assistant professor at UCSD(1993-2001). Javier has been studying learning and per-ception by human and machine for more than 20 yearsand has published over 50 articles in scientific journalsand peer-reviewed conference proceedings. His articlesspan studies in probability theory, machine learning,machine perception, experimental psychology, and devel-opmental psychology. Javier founded UCSD’s MachinePerception Laboratory in 1997 with the goal of devel-oping machine perception systems that combine multiplesources of information (e.g. audio and video) and interactnaturally with people, reading their lips. recognizingtheir facial expressions, and making inferences aboutcognitive and affective states. Javier started the Kol-mogorov project that provides a collection of open sourcetutorials on topics related to Machine Learning, MachinePerception and Statistics (http://markov.ucsd.edu/ kol-mogorov/history.html). In 2004 he chaired the 3rd In-ternational Conference on Development and Learning inSan Diego.



Automatic Recognition of Facial Actions in Spontaneous ...€¦ · The facial action coding system...

Documents

Transcript of Automatic Recognition of Facial Actions in Spontaneous ...€¦ · The facial action coding system...