Research Article Dimensional Information-Theoretic...

Research ArticleDimensional Information-Theoretic Measurement of FacialEmotion Expressions in Schizophrenia

Jihun Hamm1 Amy Pinkham2 Ruben C Gur34 Ragini Verma5 and Christian G Kohler3

1 Department of Computer Science and Engineering The Ohio State University Columbus OH 43210 USA2 Southern Methodist University Dallas TX 75205 USA3Department of Psychiatry Neuropsychiatry Section University of Pennsylvania Philadelphia PA 19104 USA4Philadelphia Veterans Administration Medical Center Philadelphia PA 19104 USA5Department of Radiology Section of Biomedical Image Analysis University of Pennsylvania Philadelphia PA 19104 USA

Correspondence should be addressed to Christian G Kohler kohlerupennedu

Received 9 July 2013 Revised 26 December 2013 Accepted 28 December 2013 Published 25 February 2014

Academic Editor Robin Emsley

Copyright copy 2014 Jihun Hamm et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Altered facial expressions of emotions are characteristic impairments in schizophrenia Ratings of affect have traditionally beenlimited to clinical rating scales and facial muscle movement analysis which require extensive training and have limitations basedon methodology and ecological validity To improve reliable assessment of dynamic facial expression changes we have developedautomated measurements of facial emotion expressions based on information-theoretic measures of expressivity of ambiguity anddistinctiveness of facial expressions These measures were examined in matched groups of persons with schizophrenia (119899 = 28) andhealthy controls (119899 = 26) who underwent video acquisition to assess expressivity of basic emotions (happiness sadness anger fearand disgust) in evoked conditions Persons with schizophrenia scored higher on ambiguity the measure of conditional entropywithin the expression of a single emotion and they scored lower on distinctiveness the measure of mutual information acrossexpressions of different emotions The automated measures compared favorably with observer-based ratings This method can beapplied for delineating dynamic emotional expressivity in healthy and clinical populations

1 Introduction

The ability to communicate emotions through facial expres-sions is crucial for interpersonal engagement Altered facialexpressions of emotions are viewed as characteristic impair-ments in schizophrenia [1 2] that already feature prominentlyduring early illness [3 4] and may precede the onset ofpsychosis [5] Assessment of facial expressions has tradi-tionally relied on observer-based rating scales The Scale forthe Assessment of Negative Symptoms (SANS [2]) and thePositive and Negative Syndrome Scale (PANSS [6]) are com-monly used by clinicians tomeasure facial expressions duringinterview settings including the ratings of affective flatteningand inappropriate affect The limited range of quantitativescores obtained during clinical interviews that preferentially

focus on positive symptoms without standard prompts toelicit changes in expressivity decrease the ecological validityof facial expression ratings

An alternative line of investigation of facial expressionsof emotions has moved away from clinical assessment ofexpressions and examined local facial changes based onmuscle movements or qualitative assessment of dynamicchanges associated with facial expressions The Facial ActionCoding System (FACS [7 8]) provides detailed descrip-tions of individual facial muscle movements in multiplefacial regions FACS and some of its adaptations have beenapplied to schizophrenia [9ndash13] However the FACS pro-cedure requires extensive training and is time-consumingwhich makes it difficult to perform in large-scale studiesDue to the prohibitive amount of time to rate videos the

Hindawi Publishing CorporationSchizophrenia Research and TreatmentVolume 2014 Article ID 243907 10 pageshttpdxdoiorg1011552014243907

2 Schizophrenia Research and Treatment

FACS procedure is usually performed for static emotionaldisplays The Facial Expression Coding System (FACES[14]) can classify dynamic facial expressions with detailssuch as valence type and intensity of emotions as deter-mined by a rater FACES ratings of videos of evokedemotions in schizophrenia have shown significant flatnessand inappropriateness during spontaneous expressions [1516] as well as for simulated [17] and evoked expressions[18]

Media for capturing facial expressions have included stillphotographs [18ndash20] videotapes [9ndash13 15 17 21 22] andelectromyographic recordings [23ndash25] Videotaped acquisi-tion offers the advantage of capturing duration and fre-quency of emotion expressions However analysis of suchlengthy data sets has been limited to more global assessmentof positive and negative emotion expressions rather thanchanges in specific face regions Common measurementsof emotion expressions have included recognition rates ofexpressions [19 21 26ndash28] and FACS-derived measureswithout analysis of specific AUs [9ndash12 15 17] Other methodshave included computerized face morphometry [25 27]and electromyographic measurements [23ndash25 29 30] thatcan measure minute muscle activations albeit limited toselect face regions for example zygomatic and corrugatormuscles that are typically utilized in frowning and smilingexpressions An interesting and frequent finding arising fromthese investigations has been the possible disconnectednessbetween emotion expressions and subjective emotion experi-ence [18 30 31] which in turn may affect social interactionsin schizophrenia

In an effort to derive more objective and scalable mea-surement of facial expressivity we have pursued computer-ized measurement of facial expressions that were obtainedduring standardized procedures to elicit emotion expres-sions Initial efforts based on manually defined face regionsfrom still images in healthy actors [32] and persons withschizophrenia [33] showed significant group differences inexpressions of happiness sadness anger and fear Wanget al [34] refined the method to allow automated analysisof videos and demonstrated different frequencies of fouremotions in three exemplar participants To account for lowintensity and ambiguous expressions of emotions Hammet al [35] proposed automated measurement of individualfacial muscles using FACS parameters and demonstratedthe differences of facial muscle activities in four patientsand controls However the implications of altered individualfacial muscle movements on global expressivity and respec-tive differences in schizophrenia where facial expressionsare commonly impaired remain to be better elucidated Inthis paper we present novel measures of facial expressivitybased on individual muscle movements that are derivedfrom information theory [36] and applied these measuresto matched groups of schizophrenia and control subjectsInformation theory originated in the 1950s from the fieldof applied mathematics to analyze the capacity of com-munication channels The method has provided powerfultools to analyze human performance measures in psycho-logical experiments [37] and a framework for understandingemotional communications [38] Information theory has

also been successfully used in biomedical investigationssuch as processing capacity of neural coding [39] com-plexity in electrocardiogram sequences [40] and electroen-cephalography in Alzheimerrsquos disease [41] and schizophrenia[42]

As an extension of our previous work that examinedboth observer-based and computerized measurements offacial expressivity of emotions we tested sensitivity ofthe information-theoretic measures in characterizing andquantifying affective expression deficits in schizophreniaand we examined their relationship with observer-basedratings of inappropriate and flattened emotion expressionsWe applied two-dimensional measures of facial expressivitythat can be computed objectively from videos of facialexpressions without requiring observer-based ratings (1)ambiguity of facial expressions within a single emotionand (2) distinctiveness of facial expressions across separateemotions These measures correspond to the two mostimportant information-theoretic quantities of (1) conditionalentropy and (2) mutual information Briefly ambiguity is theamount of uncertainty in a personrsquos facial muscle patternsduring expression of a single emotion as contrasted withconsistency of the pattern A person whose facial musclepattern is only brief or is variable during emotion expressionwill be less effective in communicating his or her intendedemotion to another person Distinctiveness is the capacity ofa person to express different emotions succinctly throughfacial muscles A person who is unable to produce distinctfacial patterns for different emotions will also be less effectivein communicating a specific emotion We anticipated thatambiguity and distinctiveness measures can be applied tolarge data sets of dynamic expressions and they can cap-ture aspects of expressivity deficits that would improve ourunderstanding of emotion expression abilities in personswithschizophrenia In addition although representing differenttheoretical constructs we examined whether information-theoretic measures correlate with observer-based ratingssuch as inappropriate and flattened affect

2 Methods

21 Subjects We collected videos of healthy controls andpersons with schizophrenia for a neuropsychiatric study ofemotions under an approved IRB protocol at the Universityof Pennsylvania After describing the study to the subjectswe obtained written informed consents including consent topublish pictures There were 28 outpatients with a DSM-IVdiagnosis of schizophrenia and 26 healthy controls balancedin gender race age and parental education All patients wereclinically stable without hospitalization for at least 3 monthsprior to research assessment and had been maintained ontheir present medication for the past month Presence ofsignificant acute extrapyramidal symptoms as evidenced bya score of 1 or higher on at least 2 of the rigidity or tremoritems (items 2 3 4 5 and 9) was exclusionary Likewisepresence of significant tardive extrapyramidal symptoms asevidenced by a score of 2 or higher on items 1ndash4 (facial andoral movements) was exclusionary All patients were treated

Schizophrenia Research and Treatment 3

Table 1 Demographic and clinical characteristics

Patient group Control groupGender (M F) 14 14 16 10Ethnicity

(Caucasian African-American other) 9 16 3 8 15 3Age

Mean 323 plusmn 90 330 plusmn 86

Range 20ndash47 yrs 20ndash48 yrsDuration of illness 1300 plusmn 83 NA

Range 2ndash33 NASymptoms

Positive (SAPS-total) 600 plusmn 85 NANegative (SANS-total) 1300 plusmn 83 NA

AntipsychoticsChlorpromazine-equivalents (119899 = 28) 225mg plusmn 247 NAOlanzapine-equivalents (119899 = 2) 136mg plusmn 76 NA

Imagevideo Feature extraction Action unit likelihoods

AU1AU2AU4AU5AU6AU7AU9AU10AU12AU15AU17AU18AU20AU23AU25

Figure 1 Automated Action Unit recognition system For each frame of a video geometric changes in facial components (red curves) areautomatically tracked and textural changes due to temporarywrinkles are detected inmultiple regions of interest (blue boxes)These extractedfeatures are fed through pattern classifiers to yield activity of each Action Unit (green bars longer bar means higher activity)

with second-generation antipsychotics that were converted todosages equivalent to olanzapine (OLZ) two patients werealso treated with first-generation antipsychotics that wereconverted to dosages equivalent to chlorpromazine (CPZ)All medication dosages were stable for the past month priorto testing and no patient was treated with anticholinergicmedications Pertinent demographic and clinical informationis summarized in Table 1

22 Emotion Elicitation Procedure and Video Acquisition Totest emotion expression ability we followed the emotion elici-tation procedure previously described [43] and adapted it foruse in schizophrenia [13] Videos were obtained for neutralexpressions and for five universal emotions (happiness sad-ness anger fear and disgust) Before recording participantswere asked to describe biographical emotional situationswhen each emotion was experienced in mild moderate andhigh intensities and these situations were summarized as

vignettes Subsequently subjects were seated in a brightly litroom where recordings took place and emotional vignetteswere recounted to participants in a narrative manner usingexact wording derived from the vignettes The spontaneousfacial expressions of the subjects were recorded as videosBefore and between the five emotion sessions the subjectswere asked to relax and return to a neutral stateThe durationof each session was about two minutes (110 plusmn 54 sec)

23 Video Processing and Automated FACS Our group hasdeveloped an automated facial coding procedure that canscore intensity of facialmuscle activity known asActionUnits(AUs [7 8]) by computerized analysis of videos The detailsof the system and the validation of accuracy appeared inHamm et al [35] and here we describe key components ofthe system and how we used it to measure information (seeFigure 1) Videos acquired in evoked emotions were analyzedframe by frame For each frame geometric changes in facial


components such as eyes eyebrows nasolabial line andlips were automatically tracked and textural changes due totemporary wrinkles were detected in multiple regions of theface To increase the reliability of the tracking a small fraction(sim3) of frames from videos in which the face leaves thescreen (ie nonfrontal) were automatically discarded fromanalysis Extracted geometric and texture features were thenpassed through pattern classifiers to yield intensity of thefollowing 15 AUs AU1 (Inner BrowRaiser) AU2 (Outer BrowRaiser) AU4 (Brow Lowerer) AU5 (Upper Lid Raiser) AU6(Cheek Raise) AU7 (Lid Tightener) AU9 (Nose Wrinkler)AU10 (Upper Lip Raiser) AU12 (Lip Corner Puller) AU15(Lip Corner Depressor) AU17 (Chin Raiser) AU18 (LipPuckerer) AU20 (Lip Stretcher) AU23 (Lip Tightener) andAU25-27 (Lips Part and Mouth Open) The other AUs thanthese 15 AUs were not used since they were too infrequentlyin the recorded videos

24 Computation of Ambiguity and Distinctiveness of FacialExpressions From the distribution of 15-dimensional con-tinuous variables collected from videos we computedinformation-theoretic measures of expressivity for each sub-ject Specifically we measure ambiguity of facial expressionpatterns within each emotion and distinctiveness of facialexpression patterns across emotions These two measurescorrespond to conditional entropy and mutual informationwhich are the fundamental variables of information the-ory [36] Interpretations of the two information-theoreticmeasures depend on the experiments being conductedand ambiguity and distinctiveness are specific interpreta-tions of our experimental results with the spontaneousexpression of emotions In psychology the two measureshave been called equivocation and information transmis-sion respectively in the context of absolute judgment tasks[44]

Computation of ambiguity and distinctiveness requiresestimation of differential entropy from facial muscles Dif-ferential entropy is the amount of uncertainty in continuousprobabilistic distributions from which we derive mutualinformation and conditional entropy Let 119909 denote the (dis-crete) emotional state of an individual and 119910 denote the(continuous and multivariate) facial muscle activity Differ-ential entropy is then defined by119867(119901(119910)) = 119864[minus log119901(119910)] =minus int

R119889119901(119910) log119901(119910)119889119910 Unlike discrete entropy differential

entropy is a relative measure and can have negative values aunivariate Gaussian distribution with 120590 = (2120587119890)minus12 has zerodifferential entropy and Gaussians with narrower or widerpeaks have negative or positive entropy respectively Since wedo not know 119901(119910) a priori the differential entropy has to beestimated from samples much like estimation of mean andvariance Several estimation methods have been proposedincluding adaptive partitioning kernel density estimationand nearest neighbor estimation (see Beirlant et al [45] fora review) We used a k-nearest neighbor estimator of entropy[46ndash48]

119867(119901(119910)) asymp (119889119899)sum119899

119894

log 120588 + V119889

minus 120595(119896) + log 119899The V119889

=

1205871198892

Γ(1198892+1) is the volume of a d-dimensional unit sphere

where Γ is the gamma function Γ(119911) = int

infin

0

119905119911minus1

119890minus119905

119889119905 and120595(119911) is the digamma function 120595(119911) = Γ(119911)

1015840

Γ(119911) The onlyfree parameter of nearest-neighbor estimate is the size of theneighbors 119896 for which we used the heuristic rule [47]

k = round (11989912 + 05) where 119899 is the number of samplesFor numerical stability we also added negligible amount (=10minus8

) of random noise to the data while computing entropyIn our experiments the conditional entropy is defined as

119867(119910 | 119909) = sum119909=119909119894

119901(119909119894

)119867(119910 | 119909119894

) = sum119909=119909119894

119901(119909119894

) int minus119901(119910 |

119909119894

) log(119901(119910 | 119909119894

))119889119910 which is the average entropy of facialexpression per emotion computed from facial muscle activity119910 in the video of each emotion 119909 with equal priors for eachemotion (119901(119909) = 15) We refer to this conditional entropy asambiguity in the following context when an individualrsquos facialexpression is consistent within each emotion the conditionalentropy is low and when the expression is varying andambiguous within each emotion the conditional entropy ishigh

The mutual information 119868(119909 119910) can be computed from119868(119909 119910) = 119867(119910) minus 119867(119910 | 119909)Mutual information between dis-crete and continuous variables as in our case is also knownas Jensen-Shannon divergence [49 50] and is nonnegativeand bounded By reformulating 119868(119909 119910) as the average KL-divergence [51] between conditional 119901(119910 | 119909) and marginal119901(119910) distributions 119868(119909 119910) = 119870119871(119901(119909 119910)119901(119909)119901(119910)) =

sum119909

119901(119909) int 119901(119910 | 119909) log(119901(119910119909)119901(119910))119889119910 = sum119909

119901(119909)119870119871(119901(119910 |

119909)119901(119910)) we notice that mutual information measures theaverage distance of emotion-specific facial expression pattern(119901(119909 | 119910)) from the patterns of all emotions combined (119901(119909))Hence our choice of the term distinctiveness is for mutualinformation

25 Interpretation of Ambiguity and Distinctiveness MeasuresIn this paper we report the ambiguity and the distinctivenessas z-scores (= (119909 minus 119898)119904) instead of their raw values foran easier interpretation where the mean and the standarddeviation are computed using all subjects and conditionsWedo this because these values are unitless and dependent on theexperimental setting and therefore meaningful only in a rela-tive sense For example if we use six basic emotions includingldquosurpriserdquo instead of five the absolute values of ambiguityand distinctiveness will change However the difference ofvalues in diagnostic groups or conditions still measures therelative amount of the ambiguity and distinctiveness of facialexpressions and provides meaningful information Note thatthe standardization of raw values using z-scores does notaffect the subsequent statistical analysis

Lastly the ambiguity and the distinctiveness are com-puted across all emotions and not for individual emotionsWhile it is possible to analyze ambiguity for each emotionpooling the values across emotions results in more reliablemeasures

26 Observer-Based Measures

Validation of Ambiguity and Distinctiveness To verify thatthe information-theoretic measures agree with an observerrsquosinterpretation of ambiguity and distinctiveness from videos


the following criteria for manual scores from 0 to 4 weredefined and each video was rated by an observer blind tothe diagnosis of subjects For ambiguity which was ratedfor videos of each emotion 0 meant very consistent (withonly single facial expression pattern in the single videoof an emotion) and 4 meant very ambiguous (with morethan four different facial expression patterns in the videoof an emotion) Scores 1 to 3 corresponded to intermediatelevels of ambiguity (with 1 2 and 3 major facial expressionpatterns in a video resp) For distinctiveness which was ratedacross videos of five emotions of a single subject 0 meantthe five emotional videos were totally indistinguishable inrepresenting the target emotions and 4 meant all five videoswere distinctive and representative of the target emotionsScores 1 to 3 corresponded to intermediate levels of dis-tinctiveness (with 1 2 and 3 videos of distinctive emotionsresp)

Observer-Based Ratings of Facial Expressions To compareinformation measures with previous observer-based ratingsflatness and inappropriateness of facial expression from theScale for the Assessment of Negative Symptoms (SANS)were adapted to video-based ratings [33] Two raters scoredeach video with separate scores for flat and inappropri-ate affect ranging from 0 (none) to 4 (extremely flat orinappropriate) SANS raters knew the intended emotionof the video but not the diagnosis of subjects Video-based SANS was based on a 5-point rating similar toobserver-based SANS Ratings that differed by 2 or morepoints were reviewed for consensus and final ratings wereaveraged

27 Statistical Analysis The following data analysis wasperformed to demonstrate applicability and validity of themeasures of ambiguity and distinctivenessGroup-based com-parisons for ambiguity and distinctiveness of expressionsinvolved were carried out via two-way ANOVA of ambiguityand distinctiveness separately using sex and diagnosis asgrouping factors We also measured the effect size of diagno-sis by Cohenrsquos d (= (119898

1

minus 1198982

)(05(1199042

1

+ 1199042

2

))

12

) Validationof computerized measures of ambiguity and distinctivenessagainst observer-based measures was performed by Pearsoncorrelations where higher values meant better agreementbetween the computerized and observer-basedmeasures Forobserver-based ratings of inappropriateness and flatness ofexpressions we performed separate two-way ANOVA usingsex and diagnosis as grouping factors We also measured theeffect size of diagnosis by Cohenrsquos d For the purposes ofinterpretability and comparison with alternative measures ofsymptom severity we performedmultiple regression analysisto study the explanatory power of the computerizedmeasuresof ambiguity and distinctiveness for observer-based ratingsof inappropriate and flat affects in which computerizedmeasures were used as independent variables and each ofthe observer-based ratings was used as a dependent variableWe performedmultiple regression also in the other directionin which observer-based ratings were used as independent

variables and each of the computerized measures was usedas a dependent variable

3 Results

31 Ambiguity and Distinctiveness of Emotion ExpressionsAmbiguity of expression averaged across emotions showed astrong effect of diagnosis (119865 = 12 119889119891 = 1 53 119875 = 00012)but no effect of sex (119865 lt 0001 119889119891 = 1 53 119875 = 099)

nor interaction (119865 = 078 119889119891 = 1 53 119875 = 038) by two-way ANOVA Patients showed higher ambiguity (041 plusmn 10)than controls (minus044 plusmn 079) with a large effect size of 093(Cohenrsquos d) Likewise distinctiveness of expression showed astrong effect of diagnosis (119865 = 83 119889119891 = 1 53 119875 = 00057)but no effect of sex (119865 = 0056 119889119891 = 1 53 119875 = 081) norinteraction (119865 = 026 119889119891 = 1 53 119875 = 061) by two-wayANOVA Patients showed lower distinctiveness (minus037plusmn088)than controls (040 plusmn 098) with a large effect size of Cohenrsquosd = 083

The characteristics of ambiguity and distinctiveness mea-sures are demonstrated with sample videos of subjects inFigures 2ndash5 that illustrate the expressions of subjects withratings for low ambiguityhigh distinctiveness in contrast tosubjects with ratings for low ambiguitylow distinctivenesshigh ambiguitylow distinctiveness and high ambiguityhighdistinctiveness

32 Observer-Basted Measures

Validation of Ambiguity and Distinctiveness The computer-ized measures of ambiguity and distinctiveness were well cor-related with the observerrsquos scores of ambiguity (119903 = 071 119875 lt001) and distinctiveness (119903 = 060 119875 lt 001) Thesecorrelations supported the notion of agreement betweenthe computerized measures from information theory andthe observer-rated measures from visual examination of thevideos

Observer-Based Ratings of Facial Expressions For video-based expert SANS ratings flatness of expression showed amoderate effect of diagnosis (119865 = 46 119889119891 = 1 53 119875 =

0038) but no effect of sex (119865 = 39 119889119891 = 1 53 119875 =

0055) nor interaction (119865 = 019 119889119891 = 1 53 119875 = 066)

by two-way ANOVA Patients were more flat (20 plusmn 093)

than controls (15 plusmn 091) with a medium effect size of 057The inappropriateness of expression showed a strong effect ofdiagnosis (119865 = 96 119889119891 = 1 53 119875 = 00033) but no effectof sex (119865 = 099 119889119891 = 1 53 119875 = 033) nor interaction(119865 = 00047 119889119891 = 1 53 119875 = 095) by two-way ANOVAPatients were rated higher on inappropriate affect (11plusmn065)than controls (063 plusmn 037)with a large effect size of 092

33 Relationship between Computerized and Observer-Based Measures We examined the relationships betweeninformation-theoretic measures of ambiguity anddistinctiveness and observer-based measures of inappropriate


and flattened facial expressions of emotions Regressionof ambiguity on observer-based measures was moderatelypredictive (1198772 = 013 119865 = 34 119875 = 0040) and thecoefficients from flatness and inappropriateness were minus014(119875 = 030) and 050 (119875 = 0032) respectively Regressionof distinctiveness on observer-based measures was highlypredictive (1198772 = 019 119865 = 58 119875 = 00056) and thecoefficients from flatness and inappropriateness were minus032(119875 = 0027) and minus065 (119875 = 00063) respectively Regressionof flatness on computerized measures was moderatelypredictive (1198772 = 015 119865 = 43 119875 = 0019) and thecoefficients from ambiguity and distinctiveness were minus033(119875 = 0025) and minus034 (119875 = 0013) respectively Lastlyregression of inappropriateness on computerized measureswas moderately predictive (1198772 = 015 119865 = 43 119875 = 0018)and the coefficients from ambiguity and distinctivenesswere 014 (119875 = 011) and minus014 (119875 = 011)respectively

4 Discussion and Conclusions

Impaired abilities of facial expression of emotions are com-mon dysfunctions in schizophrenia that are associated withworse quality of life and poorer outcome [52 53] Clini-cal assessment of emotion expression abilities is typicallyobtained using observer-based rating scales administeredduring interviews that are not standardized to elicit emotionexpressions and with limited potential to compare dataacross different studies and populations More advancedobserver-basedmeasurements of facial expressions have beenchallenged by the complexity inherent in rating regionaland global changes in dynamic facial expressions and theirapplications have been mainly limited to research Neverthe-less these investigations using standardized observer basedratings and electromyography have underscored that personswith schizophrenia exhibit altered expressions of volitionaland of more fleeting spontaneous facial emotions that donot necessarily reflect their internal emotional state Anotherchallenge of measuring emotion expressions has been theimportance to obtain emotion expressions that are genuineand naturalistic and not obscured by the artifice of the testingsetting and methodology or influenced by rater fatigueThis is where automated computerized measurement offersan advantage over other methods to investigate dynamicexpressions

Our group has developed computerized assessment ofemotion expression abilities that allows for objective mea-surement of facial expressions that are obtained in a standard-ized setting [32ndash35] As an extension to these efforts we haveapplied computerized assessments of facial expressivity topatients with schizophrenia and healthy controls to examinewhether the novel methodology can elucidate differences infacial expression of emotions Computerized information-theoretic measures focus on differences in ambiguity whichrepresents the measure of variability within the expressionof a single emotion and distinctiveness which represents

the measure of how distinctive facial expressions are for aparticular emotion in comparison to other emotions Thecomputerized approach was validated by observer-basedratings of ambiguity and distinctiveness In addition whencomputerized measures were predicted by observer-basedratings ambiguitywas positively related to inappropriatenessand distinctiveness was negatively related to both flatnessand inappropriateness Conversely when observer-basedratings were predicted by computerized measures flatnesswas negatively related to both ambiguity and distinctivenesswhile inappropriateness was not significantly related witheither measure alone although it was significantly pre-dicted by overall ambiguity and distinctiveness As illustratedin Figures 2ndash5 computerized measures of ambiguity anddistinctiveness were associated with facial expressions thatin the combination of low ambiguityhigh distinctiveness(Figure 2) were well recognizable within each emotion andalso different across emotionsHigh ambiguitywith either low(Figure 3) or high (Figure 4) distinctiveness values appearedas inappropriate expressions that were either similar ordifferent across emotions The computerized measures alsoindicate flatness as when facial expressions are not vari-able within an emotion (low ambiguity) and also indistin-guishable across emotions (low distinctiveness) (Figure 5)These results suggest that the information-theoretic mea-sures of emotion expressions are related to the observer-based ratings and computerized measures may providequantifiable information on different causes of observedinappropriate affect Specifically inappropriateness of expres-sion may result either from ambiguous and inconsistentfacial expressions in each emotion regardless of distinctive-ness across emotions Flat affect on the other hand wasmainly related to emotion expressions being indistinct acrossemotions

Our computerized method employing information-theoretic measures offers several methodological advantagesover currently available observer-based rating scales andFACS-based rating instruments for example our methodbeing more objective and repeatable for any number ofsubjects at the same time being less labor intensive and time-consuming It ismore objective than FACES since themethoddoes not involve observer-judgment of emotions expressedand is less labortime intensive than FACS and FACES since itis fully automated similar to EMGwithout the inconvenienceof physically placing electrodes on the participantsrsquofaces

While we demonstrated the feasibility and applicabilityof these measures with schizophrenia patients and healthycontrols we are applying the analysis to a larger sample thatcan better reflect the range of individual differences in therespective populations A major limitation of our approachis based on the fact that no computerized determinationwas made regarding emotional valence and such qualita-tive information could have enhanced the results basedon information-theoretic measures of expressivity Anotherlimitation of the current study is that the induction procedurewithin the laboratory while standardizedmay lack ecological


H

S

A

F

D

Figure 2 Sample videos of a subject with low ambiguity and high distinctiveness H S A F and D stand for happiness sadness angerfear and disgust Each row shows sample images evenly chosen from a video of the emotion Facial expressions of low ambiguity and highdistinctiveness are consistent within each emotion and also different across emotions which makes each emotion well recognizable

H

S

A

F

D

Figure 3 Sample videos of a subject with high ambiguity and low distinctiveness High ambiguity and low distinctiveness appear asinappropriate facial expressions due to the lack of consistent patterns within each emotion and the overlap of patterns across emotions

validity A more naturalistic setting would include film-ing participants while they recount their own experiencesUnfortunately at the present state of this methodology facialmovements on account of speech would generate too muchnoise for our algorithms to overcome

In conclusion ourmethod offers an automatedmeans forquantifying individual differences in emotional expressivitysimilar to what has been accomplished in the area of emotionrecognitionThis capability opens new venues for delineatingemotional expressivity in healthy people and across clinical


H

S

A

F

D

Figure 4 Sample videos of a subject with high ambiguity and high distinctiveness High ambiguity and high distinctiveness also appear asinappropriate facial expressions due to the lack of consistent patterns within each emotion

H

S

A

F

D

Figure 5 Sample videos of a subject with low ambiguity and low distinctiveness Facial expressions of low ambiguity and low distinctivenesspresent themselves as muted or flat expressions throughout all emotions

populations As for investigations in healthy population theautomated procedure is well suited to examine both ageand gender related changes in emotion expression abilitiesPotential clinical applications may include repeated moni-toring of facial expressions to investigate effects of disease

progression andmore general treatment effects or tomeasuretargeted efforts to remediate emotion expression abilitiesAutomated comparisons need not be limited to select clinicalpopulations and may allow for effective comparisons ofemotion expressivity across different psychiatric disorders


Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This work was supported by the National Institutes of Health(R01-MH073174 and R01-MH060722) and by Stanley Foun-dationGrantThe authors thank ElizabethMartin andKristinHealey for their assistance in data acquisition

References

[1] E Bleuler ldquoDementia praecox oder die gruppe der schizophre-nienrdquo 1911

[2] N C Andreasen The Scale for the Assessment of NegativeSymptoms (SANS) The Scale for the Assessment of PositiveSymptoms (SAPS) The University of Iowa Iowa City IowaUSA 1984

[3] A KMalla J J Takhar RMGNorman et al ldquoNegative symp-toms in first episode non-affective psychosisrdquo Acta PsychiatricaScandinavica vol 105 no 6 pp 431ndash439 2002

[4] J Makinen J Miettunen M Isohanni and H Koponen ldquoNeg-ative symptoms in schizophreniamdasha reviewrdquo Nordic Journal ofPsychiatry vol 62 no 5 pp 334ndash341 2008

[5] E F Walker K E Grimes D M Davis and A J SmithldquoChildhood precursors of schizophrenia facial expressions ofemotionrdquo American Journal of Psychiatry vol 150 no 11 pp1654ndash1660 1993

[6] S R Kay A Fiszbein and L A Opler ldquoThe positive and nega-tive syndrome scale (PANSS) for schizophreniardquo SchizophreniaBulletin vol 13 no 2 pp 261ndash276 1987

[7] P Ekman and W Friesen Facial Action Coding System Investi-gators Guide Consulting Psychologists Press Palo Alto CalifUSA 1978

[8] P Ekman and W Friesen Manual of the Facial Action CodingSystem (FACS) Consulting Psychologists Press Palo Alto CalifUSA 1978

[9] H Berenbaum ldquoPosed facial expressions of emotion inschizophrenia and depressionrdquo Psychological Medicine vol 22no 4 pp 929ndash937 1992

[10] H Berenbaum and T F Oltmanns ldquoEmotional experienceand expression in schizophrenia and depressionrdquo Journal ofAbnormal Psychology vol 101 no 1 pp 37ndash44 1992

[11] W Gaebel and W Wolwer ldquoFacial expressivity in the course ofschizophrenia and depressionrdquo European Archives of Psychiatryand Clinical Neuroscience vol 254 no 5 pp 335ndash342 2004

[12] F Tremeau D Malaspina F Duval et al ldquoFacial expressivenessin patients with schizophrenia compared to depressed patientsand nonpatient comparison subjectsrdquo American Journal ofPsychiatry vol 162 no 1 pp 92ndash101 2005

[13] C G Kohler E A Martin N Stolar et al ldquoStatic posedand evoked facial expressions of emotions in schizophreniardquoSchizophrenia Research vol 105 no 1ndash3 pp 49ndash60 2008

[14] A M Kring and D M Sloan ldquoThe facial expression codingsystem (FACES) development validation and utilityrdquo Psycho-logical Assessment vol 19 no 2 pp 210ndash224 2007

[15] A M Kring S L Kerr D A Smith and J M Neale ldquoFlataffect in schizophrenia does not reflect diminished subjective

experience of emotionrdquo Journal of Abnormal Psychology vol102 no 4 pp 507ndash517 1993

[16] A M Kring and J M Neale ldquoDo schizophrenic patientsshow a disjunctive relationship among expressive experientialand psychophysiological components of emotionrdquo Journal ofAbnormal Psychology vol 105 no 2 pp 249ndash257 1996

[17] M A Aghevli J J Blanchard andW P Horan ldquoThe expressionand experience of emotion in schizophrenia a study of socialinteractionsrdquo Psychiatry Research vol 119 no 3 pp 261ndash2702003

[18] CGKohler E AMartinMMilonova et al ldquoDynamic evokedfacial expressions of emotions in schizophreniardquo SchizophreniaResearch vol 105 no 1ndash3 pp 30ndash39 2008

[19] E Gottheil C C Thornton and R V Exline ldquoAppropriate andbackground affect in facial displays of emotion Comparison ofnormal and schizophrenic malesrdquo Archives of General Psychia-try vol 33 no 5 pp 565ndash568 1976

[20] B L Schwartz J Mastropaolo R B Rosse G Mathis and SI Deutsch ldquoImitation of facial expressions in schizophreniardquoPsychiatry Research vol 145 no 2-3 pp 87ndash94 2006

[21] K M Putnam and A M Kring ldquoAccuracy and intensity ofposed emotional expressions in unmedicated schizophreniapatients vocal and facial channelsrdquo Psychiatry Research vol 151no 1-2 pp 67ndash76 2007

[22] E Steimer-Krause R Krause and G Wagner ldquoInteractionregulations used by schizophrenic and psychosomatic patientsstudies on facial behavior in dyadic interactionsrdquo Psychiatryvol 53 no 3 pp 209ndash228 1990

[23] K S Earnst A M Kring M A Kadar J E Salem D AShepard and P T Loosen ldquoFacial expression in schizophreniardquoBiological Psychiatry vol 40 no 6 pp 556ndash558 1996

[24] A M Kring S L Kerr and K S Earnst ldquoSchizophrenicpatients show facial reactions to emotional facial expressionsrdquoPsychophysiology vol 36 no 2 pp 186ndash192 1999

[25] R M Mattes F Schneider H Heimann and N BirbaumerldquoReduced emotional response of schizophrenic patients inremission during social interactionrdquo Schizophrenia Researchvol 17 no 3 pp 249ndash255 1995

[26] E Gottheil A Paredes R V Exline and R WinkelmayerldquoCommunication of affect in schizophreniardquoArchives of GeneralPsychiatry vol 22 no 5 pp 439ndash444 1970

[27] F Schneider H Heimann W Himer D Huss R Mattesand B Adam ldquoComputer-based analysis of facial action inschizophrenic and depressed patientsrdquo European Archives ofPsychiatry and Clinical Neuroscience vol 240 no 2 pp 67ndash761990


[29] KWolf RMass F Kiefer KWiedemann andDNaber ldquoChar-acterization of the facial expression of emotions in schizophre-nia patients preliminary findings with a new electromyographymethodrdquo Canadian Journal of Psychiatry vol 51 no 6 pp 335ndash341 2006

[30] M Iwase K Yamashita K Takahashi et al ldquoDiminished facialexpression despite the existence of pleasant emotional experi-ence in schizophreniardquo Methods and Findings in Experimentaland Clinical Pharmacology vol 21 no 3 pp 189ndash194 1999

[31] C E Sison M Alpert R Fudge and R M Stern ldquoCon-stricted expressiveness and psychophysiological reactivity inschizophreniardquo Journal of Nervous amp Mental Disease vol 184no 10 pp 589ndash597 1996


[32] R Verma C Davatzikos J Loughead et al ldquoQuantification offacial expressions using high-dimensional shape transforma-tionsrdquo Journal of NeuroscienceMethods vol 141 no 1 pp 61ndash732005

[33] C Alvino C Kohler F Barrett R E Gur R C Gur and RVerma ldquoComputerized measurement of facial expression ofemotions in schizophreniardquo Journal of Neuroscience Methodsvol 163 no 2 pp 350ndash361 2007

[34] P Wang F Barrett E Martin et al ldquoAutomated video-basedfacial expression analysis of neuropsychiatric disordersrdquo Jour-nal of Neuroscience Methods vol 168 no 1 pp 224ndash238 2008

[35] J Hamm C G Kohler R C Gur and R Verma ldquoAutomatedfacial action coding system for dynamic analysis of facialexpressions in neuropsychiatric disordersrdquo Journal of Neuro-science Methods vol 200 no 2 pp 237ndash256 2011

[36] C E Shannon ldquoAmathematical theory of communicationrdquo BellSystem Technical Journal vol 27 no 4 Article ID 379423 pp623ndash656 1948

[37] W R Garner Uncertainty and Structure as Psychological Con-cepts John Wiley amp Sons New York NY USA 1962

[38] A TDittman InterpersonalMessages of Emotion Springer NewYork NY USA 1972

[39] A Borst and F E Theunissen ldquoInformation theory and neuralcodingrdquo Nature Neuroscience vol 2 no 11 pp 947ndash957 1999

[40] M Costa A L Goldberger and C K Peng ldquoMultiscale entropyanalysis of biological signalsrdquo Physical Review E vol 71 no 2part 1 Article ID 021906 2005

[41] J Jeong J C Gore and B S Peterson ldquoMutual informationanalysis of the EEG in patients with Alzheimerrsquos diseaserdquoClinical Neurophysiology vol 112 no 5 pp 827ndash835 2001

[42] S H Na S H Jin S Y Kim and B J Ham ldquoEEG inschizophrenic patients mutual information analysisrdquo ClinicalNeurophysiology vol 113 no 12 pp 1954ndash1960 2002

[43] R C Gur R Sara M Hagendoorn et al ldquoA method for obtain-ing 3-dimensional facial expressions and its standardization foruse in neurocognitive studiesrdquo Journal of NeuroscienceMethodsvol 115 no 2 pp 137ndash143 2002

[44] W R Garner and H W Hake ldquoThe amount of informationin absolute judgmentsrdquo Psychological Review vol 58 no 6 pp446ndash459 1951

[45] J Beirlant E J Dudewicz L Gyoerfi and E C MeulenldquoNonparametric entropy estimation an overviewrdquo InternationalJournal ofMathematical and Statistical Sciences vol 6 pp 17ndash391997

[46] L F Kozachenko and N N Leonenko ldquoSample estimate of theentropy of a random vectorrdquo Problemy Peredachi Informatsiivol 23 no 2 pp 9ndash16 1987

[47] M N Goria N N Leonenko V V Mergel and P L NInverardi ldquoA new class of random vector entropy estimatorsand its applications in testing statistical hypothesesrdquo Journal ofNonparametric Statistics vol 17 no 3 pp 277ndash297 2005

[48] R M Mnatsakanov N Misra S Li and E J Harner ldquoKn-nearest neighbor estimators of entropyrdquoMathematical Methodsof Statistics vol 17 no 3 pp 261ndash277 2008

[49] J Lin ldquoDivergence measures based on the Shannon entropyrdquoIEEE Transactions on InformationTheory vol 37 no 1 pp 145ndash151 1991

[50] D M Endres and J E Schindelin ldquoA newmetric for probabilitydistributionsrdquo IEEETransactions on InformationTheory vol 49no 7 pp 1858ndash1860 2003

[51] S Kullback and R A Leibler ldquoOn Information and SufficiencyrdquoAnnals of Mathematical Statistics vol 22 no 1 pp 79ndash86 1951

[52] B CHo PNopoulosM Flaum S Arndt andNCAndreasenldquoTwo-year outcome in first-episode schizophrenia predictivevalue of symptoms for quality of liferdquo American Journal ofPsychiatry vol 155 no 9 pp 1196ndash1201 1998

[53] R E Gur C G Kohler J D Ragland et al ldquoFlat affect inschizophrenia relation to emotion processing and neurocogni-tive measuresrdquo Schizophrenia Bulletin vol 32 no 2 pp 279ndash287 2006

Submit your manuscripts athttpwwwhindawicom

Stem CellsInternational

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014


MEDIATORSINFLAMMATION

of


Behavioural Neurology

EndocrinologyInternational Journal of



Disease Markers


BioMed Research International

OncologyJournal of



Oxidative Medicine and Cellular Longevity


PPAR Research

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Immunology ResearchHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Journal of

ObesityJournal of



Computational and Mathematical Methods in Medicine

OphthalmologyJournal of


Diabetes ResearchJournal of



Research and TreatmentAIDS


Gastroenterology Research and Practice


Parkinsonrsquos Disease

Evidence-Based Complementary and Alternative Medicine

Volume 2014Hindawi Publishing Corporationhttpwwwhindawicom


FACS procedure is usually performed for static emotionaldisplays The Facial Expression Coding System (FACES[14]) can classify dynamic facial expressions with detailssuch as valence type and intensity of emotions as deter-mined by a rater FACES ratings of videos of evokedemotions in schizophrenia have shown significant flatnessand inappropriateness during spontaneous expressions [1516] as well as for simulated [17] and evoked expressions[18]

Media for capturing facial expressions have included stillphotographs [18ndash20] videotapes [9ndash13 15 17 21 22] andelectromyographic recordings [23ndash25] Videotaped acquisi-tion offers the advantage of capturing duration and fre-quency of emotion expressions However analysis of suchlengthy data sets has been limited to more global assessmentof positive and negative emotion expressions rather thanchanges in specific face regions Common measurementsof emotion expressions have included recognition rates ofexpressions [19 21 26ndash28] and FACS-derived measureswithout analysis of specific AUs [9ndash12 15 17] Other methodshave included computerized face morphometry [25 27]and electromyographic measurements [23ndash25 29 30] thatcan measure minute muscle activations albeit limited toselect face regions for example zygomatic and corrugatormuscles that are typically utilized in frowning and smilingexpressions An interesting and frequent finding arising fromthese investigations has been the possible disconnectednessbetween emotion expressions and subjective emotion experi-ence [18 30 31] which in turn may affect social interactionsin schizophrenia

In an effort to derive more objective and scalable mea-surement of facial expressivity we have pursued computer-ized measurement of facial expressions that were obtainedduring standardized procedures to elicit emotion expres-sions Initial efforts based on manually defined face regionsfrom still images in healthy actors [32] and persons withschizophrenia [33] showed significant group differences inexpressions of happiness sadness anger and fear Wanget al [34] refined the method to allow automated analysisof videos and demonstrated different frequencies of fouremotions in three exemplar participants To account for lowintensity and ambiguous expressions of emotions Hammet al [35] proposed automated measurement of individualfacial muscles using FACS parameters and demonstratedthe differences of facial muscle activities in four patientsand controls However the implications of altered individualfacial muscle movements on global expressivity and respec-tive differences in schizophrenia where facial expressionsare commonly impaired remain to be better elucidated Inthis paper we present novel measures of facial expressivitybased on individual muscle movements that are derivedfrom information theory [36] and applied these measuresto matched groups of schizophrenia and control subjectsInformation theory originated in the 1950s from the fieldof applied mathematics to analyze the capacity of com-munication channels The method has provided powerfultools to analyze human performance measures in psycho-logical experiments [37] and a framework for understandingemotional communications [38] Information theory has

also been successfully used in biomedical investigationssuch as processing capacity of neural coding [39] com-plexity in electrocardiogram sequences [40] and electroen-cephalography in Alzheimerrsquos disease [41] and schizophrenia[42]

As an extension of our previous work that examinedboth observer-based and computerized measurements offacial expressivity of emotions we tested sensitivity ofthe information-theoretic measures in characterizing andquantifying affective expression deficits in schizophreniaand we examined their relationship with observer-basedratings of inappropriate and flattened emotion expressionsWe applied two-dimensional measures of facial expressivitythat can be computed objectively from videos of facialexpressions without requiring observer-based ratings (1)ambiguity of facial expressions within a single emotionand (2) distinctiveness of facial expressions across separateemotions These measures correspond to the two mostimportant information-theoretic quantities of (1) conditionalentropy and (2) mutual information Briefly ambiguity is theamount of uncertainty in a personrsquos facial muscle patternsduring expression of a single emotion as contrasted withconsistency of the pattern A person whose facial musclepattern is only brief or is variable during emotion expressionwill be less effective in communicating his or her intendedemotion to another person Distinctiveness is the capacity ofa person to express different emotions succinctly throughfacial muscles A person who is unable to produce distinctfacial patterns for different emotions will also be less effectivein communicating a specific emotion We anticipated thatambiguity and distinctiveness measures can be applied tolarge data sets of dynamic expressions and they can cap-ture aspects of expressivity deficits that would improve ourunderstanding of emotion expression abilities in personswithschizophrenia In addition although representing differenttheoretical constructs we examined whether information-theoretic measures correlate with observer-based ratingssuch as inappropriate and flattened affect

2 Methods

21 Subjects We collected videos of healthy controls andpersons with schizophrenia for a neuropsychiatric study ofemotions under an approved IRB protocol at the Universityof Pennsylvania After describing the study to the subjectswe obtained written informed consents including consent topublish pictures There were 28 outpatients with a DSM-IVdiagnosis of schizophrenia and 26 healthy controls balancedin gender race age and parental education All patients wereclinically stable without hospitalization for at least 3 monthsprior to research assessment and had been maintained ontheir present medication for the past month Presence ofsignificant acute extrapyramidal symptoms as evidenced bya score of 1 or higher on at least 2 of the rigidity or tremoritems (items 2 3 4 5 and 9) was exclusionary Likewisepresence of significant tardive extrapyramidal symptoms asevidenced by a score of 2 or higher on items 1ndash4 (facial andoral movements) was exclusionary All patients were treated























119867(119901(119910)) asymp (119889119899)sum119899

119894

log 120588 + V119889

minus 120595(119896) + log 119899The V119889

=

1205871198892



infin

0

119905119911minus1

119890minus119905


1015840




119867(119910 | 119909) = sum119909=119909119894

119901(119909119894

)119867(119910 | 119909119894

) = sum119909=119909119894

119901(119909119894

) int minus119901(119910 |

119909119894

) log(119901(119910 | 119909119894



sum119909

119901(119909) int 119901(119910 | 119909) log(119901(119910119909)119901(119910))119889119910 = sum119909

119901(119909)119870119871(119901(119910 |










1

minus 1198982

)(05(1199042

1

+ 1199042

2

))

12



3 Results








0055) nor interaction (119865 = 019 119889119891 = 1 53 119875 = 066)













H

S

A

F

D


H

S

A

F

D





H

S

A

F

D


H

S

A

F

D







Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of






































119867(119901(119910)) asymp (119889119899)sum119899

119894

log 120588 + V119889

minus 120595(119896) + log 119899The V119889

=

1205871198892



infin

0

119905119911minus1

119890minus119905


1015840




119867(119910 | 119909) = sum119909=119909119894

119901(119909119894

)119867(119910 | 119909119894

) = sum119909=119909119894

119901(119909119894

) int minus119901(119910 |

119909119894

) log(119901(119910 | 119909119894



sum119909

119901(119909) int 119901(119910 | 119909) log(119901(119910119909)119901(119910))119889119910 = sum119909

119901(119909)119870119871(119901(119910 |










1

minus 1198982

)(05(1199042

1

+ 1199042

2

))

12



3 Results








0055) nor interaction (119865 = 019 119889119891 = 1 53 119875 = 066)













H

S

A

F

D


H

S

A

F

D





H

S

A

F

D


H

S

A

F

D







Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of




















1

minus 1198982

)(05(1199042

1

+ 1199042

2

))

12



3 Results








0055) nor interaction (119865 = 019 119889119891 = 1 53 119875 = 066)













H

S

A

F

D


H

S

A

F

D





H

S

A

F

D


H

S

A

F

D







Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of

























H

S

A

F

D


H

S

A

F

D





H

S

A

F

D


H

S

A

F

D







Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of

















H

S

A

F

D


H

S

A

F

D







Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of



















Acknowledgments


References





























































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of












































of






Disease Markers



OncologyJournal of





PPAR Research



Journal of

ObesityJournal of
















Research Article Dimensional Information-Theoretic...

Documents

Transcript of Research Article Dimensional Information-Theoretic...