Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1.

Cues to Emotion: Face, Body, Brain

Bob Coyne (cs6998)

1

Outline

• Emotion in voice and brain– The voice of emotion: an FMRI study of neural responses to

angry and happy vocal expressions – (2006) Tom Johnstone et al

• Facial Expressions and emotion– http://en.wikipedia.org/wiki/Paul_Ekman– Facial Action Coding System (FACS) – Ekman & Friesen– Microexpressions - Ekman– Poker tells: The Body Language of Poker (1984) Mike Caro– 3D graphics facial expressions – FaceGen software

• www.facegen.com

2

http://en.wikipedia.org/wiki/Paul_Ekman

The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions –

(2006) Tom Johnstone et al

• Range of issues:– Emotions in general vs specific emotions– General regions of brain (left/right) vs localized– Perception vs production– Prosody vs semantic content in speech– Facial expressions vs speech– Attention vs inattention– Type of experiment, controls, and measurement

(FRMI, MEG, PET, EEG, other) – What function does vocal affect have and how to test

3

Importance of response to vocal affect

• From the earliest stages of development, infants respond to affect-laden vocal expressions from their mothers (Fernald, 1989; Fernald and Morikawa, 1993)– Why might this be?– Relevance to the 4 approaches (darwinian, jamesian…)?

• Still important/relevant after?– When? – Emoticons?

• Emotion in voice recognized as well as faces across cultures (Scherer and Wallbott, 1994)– True?

4

Early Prior work• Production/comprehension experiments

– Production vs perception. Makes a difference?– Production in Right hemisphere (Hughlings-Jackson - 1915)

• How did they know?– Neurological evidence for comprehension (Tucker et al., 1977) – Reduced affective speech comprehension in right-hemisphere-

damaged listeners (bowers 1987, Heilman et al., 1984; Peper and Irle, 1997; Ross, 1981)

• What’s missing?– Where processed? – Controlled experiments?– How else to test/identify region?

5

More recent Prior work• Specific regions for facial emotion processing (e.g.

amygdala for fear, insula for disgust)• Little prior work on localized regions for emotional speech

for specific emotions. • Propositional content vs emotional prosody

– Right prefrontal activation (PET - George et al. 1996)– EEG scalp electroencephalography Pihan et al. 1997

• Accented syllable duration and fundamental frequency• Are those correlated with emotional speech?

• Magnetoencephalography (MEG) (Imaizumi et al. 1998)– Also found some left-hemisphere processing of propositional

and prosodic content

6

Recent Prior work• non-differentiated per emotion• Selective activation• Congruence effect with facial and vocal• Experiments

– Mitchell et al. (2003) found areas of posterior middle temporal gyrus (MTG) and superior temporal sulcus (STS) that activated more when attending to affective prosody as compared with semantic content of spoken words

– Grandjean et al. (2005) and Sander et al. (2005) have reported FMRI data that revealed a region in STS that showed greater activation in response to angry speech as compared with neutral speech

– Wildgruber et al. (2005) identified a right hemisphere network consisting of posterior STS, and dorsolateral and orbitobasal prefrontal cortex that showed selective activation during an emotion recognition task. Differential activations for the five emotions were not observed.

– Ethofer et al. (2006) identified regions in the right posterior MTG and STS and bilateral inferior/middle frontal gyrus that activated more when individuals identified affective prosody than when identifying the emotional content of the spoken words. No distinction was made between responses to the different expressed emotions studied

– Pourtois et al. (2005) demonstrated an area in left MTG that showed heightened activation when congruent vocal expressions and facial expressions of happiness or fear were simultaneously presented, as compared with when only one expressive modality was presented

7

The Brain

Whole Brain Atlas Top 100 Brain Structures: http://www.med.harvard.edu/AANLIB/cases/caseM/case.html 8

Loci of Sulci and Gyri1. Central sulcus2. Precentral gyrus3. Postcentral gyrus4. Precentral sulcus5. Postcentral sulcus6. Middle frontal gyrus 7. Superior frontal gyrus8. Superior parietal lobule 9. Occipital gyri 10. Longitudinal fissure

9

Highly specialized brain function?

10

Hypotheses• The question remains, therefore, of whether specific neural regions are

more engaged in the processing of some emotions than others. • Based on the hypothesis that affiliative social vocal signals are prevalent

throughout our lives and serve a fundamental purpose in social binding we hypothesize that emotional expressions of happiness would preferentially engage parts of the temporal cortex and inferior frontal regions previously shown to be involved in the processing of affective prosody.

• A further question concerning the perception of vocal expressions of emotion is how directed attention towards or away from the expressed emotion affects the associated neural response.– How to test this?

• Activation for face versus voice – most studies on face• What types of emotional speech detected? All?

11

Experimental approach• FMRI to detect localized regions for processing happiness vs anger in speech• Why those two emotions?• Reasons

– Control for similar acoustic signal. – Common/basic. Easier to characterize.– social utility hypothesis regarding happiness

• Which theory of emotion (jamesian etc)• What was experimental setup?• Some factors

– 40 subjects– Facial stimuli consisted of 16 grayscale images of posed expressions of anger and 16

expressions of happiness, half of them females. – Vocal stimuli consisted of short phrases (dates and numbers) lasting on average 1 s, spoken

with either angry or happy prosody. – Half the participants make their decision on the basis of the facial expression, while the other

half on the basis of the vocal expression– FMRI scan during happy/angry speech

• With corresponding or opposite facial expression

12

Experiment Issues/problems?

– Brain regions necessarily responding to emotion?– Quality of audio? Affects results?– FMRI measures blood flow. Relevant to neural?– Right handed subjects. Relevant?– Faces not correlated to the speaker….– Why normalized audio amplitude?– Why no semantic emotional content?– Why short phrases?

13

FRMI

Functional MRI or functional Magnetic Resonance Imaging (fMRI) is a type of specialized MRI scan. It measures the haemodynamic response related to neural activity in the brain or spinal cord of humans or other animals. It is one of the most recently developed forms of neuroimaging. Since the early 1990s, fMRI has come to dominate the brain mapping field due to its low invasiveness, lack of radiation exposure, and relatively wide availability. 14

Experiment - results• Happy voices elicited significantly more activation than

angry voices in right anterior and posterior middle temporal gyrus (MTG), left posterior MTG and right inferior frontal gyrus.

• Attention-dependent results: In contrast with brain regions showing a main effect of vocal emotion regardless of attentional focus, a network of brain regions including the left insula, left amygdala and hippocampus, and rostral ACC responded more to happy voices than to angry voices when attending to the voice, but showed either no difference or greater activation to angry voices than to happy voices when attending to the face.

• Do these results support the social binding hypothesis?

15

Results - RT and accuracy

16

•Why longer RT for “voice” case?•Why measure RT and accuracy at all?

Results How do these graphssupport the stated results?

17

Facial Expressions and Emotion - Paul Ekman http://en.wikipedia.org/wiki/Paul_Ekman

• Found facial expressions of emotion are not culturally determined, but universal across human cultures and thus biological in origin.– How did he determine this?– Fully true? Examples?– How does this square with the four approaches: Darwinian? Jamesian? Social constructivist?

Appraisal?• Expressions he found to be universal included those indicating anger, disgust, fear,

joy, sadness, and surprise. Findings on contempt are less clear, though there is at least some preliminary evidence that this emotion and its expression are universally recognized.– examples of non-universal emotions? of culture-dependent expression of same emotion? Of

individual-dependent expression of same emotion?• Facial Action Coding System (FACS) taxonomize all human facial expression.

(Ekman and Friesen)– 48 action descriptors, which are a contraction or relaxation of one or more muscles.– http://en.wikipedia.org/wiki/Facial_Action_Coding_System– http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm

• Microexpressions – detect lying– How does this phenomenon square with the four approaches?

18

http://en.wikipedia.org/wiki/Paul_Ekman

http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm

FACS – action descriptors

1 Inner Brow Raiser -- Frontalis (pars medialis)2 Outer Brow Raiser -- Frontalis (pars lateralis)4 Brow Lowerer -- Depressor glabellae, Depressor supercilii,

Corrugator supercilii5 Upper Lid Raiser -- Levator palpebrae superioris6 Cheek Raiser -- Orbicularis oculi (pars orbitalis)7 Lid Tightener -- Orbicularis oculi (pars palpebralis)9 Nose Wrinkler -- Levator labii superioris alaeque nasi10 Upper Lip Raiser -- Levator labii superioris, caput

infraorbitalis11 Nasolabial Deepener -- Zygomaticus minor12 Lip Corner Puller -- Zygomaticus major13 Cheek Puffer -- Levator anguli oris (also known as Caninus)14 Dimpler -- Buccinator15 Lip Corner Depressor -- Depressor anguli oris (also known

as Triangularis)16 Lower Lip Depressor -- Depressor labii inferioris17 Chin Raiser -- Mentalis18 Lip Puckerer -- Incisivii labii superioris and Incisivii labii

inferioris19 Tongue Out20 Lip stretcher -- Risorius w/ platysma21 Neck Tightener -- Platysma22 Lip Funneler -- Orbicularis oris23 Lip Tightener -- Orbicularis oris24 Lip Pressor -- Orbicularis oris

24 Lip Pressor -- Orbicularis oris25 Lips part -- Depressor labii inferioris or relaxation of

Mentalis, or Orbicularis oris26 Jaw Drop -- Masseter, relaxed Temporalis and internal

pterygoid27 Mouth Stretch -- Pterygoids, Digastric28 Lip Suck -- Orbicularis oris29 Jaw Thrust30 Jaw Sideways31 Jaw Clencher -- Masseter32 Lip Bite33 Cheek Blow34 Cheek Puff35 Cheek Suck36 Tongue Bulge37 Lip Wipe38 Nostril Dilator39 Nostril Compressor41 Lid Droop42 Slit43 Eyes Closed -- Relaxation of Levator palpebrae superioris;

Orbicularis oculi (pars palpebralis)44 Squint45 Blink -- Relaxation of Levator palpebrae superioris;

Orbicularis oculi (pars palpebralis)46 Wink -- Relaxation of Levator palpebrae superioris;

Orbicularis oculi (pars palpebralis)

19

Facial Action Coding System - FACS

FACS can be used to distinguish two types of smiles: •insincere and voluntary Pan American smile:

contraction of zygomatic major alone•sincere and involuntary Duchenne smile:

contraction of zygomatic major and inferior part of orbicularis oculi. 20

Pictures of Facial Affect (resource) – Ekman and Friesen

• 110 pictures of posed facial emotions– 10 second display to subjects (male and female 50%

mix)– 70% accurately identified

• Mark each example as one of:– Happy– Sad– Fear– Anger– Surprise– Disgust

21

89•Happy•Sad•Fear•Anger•Surprise•Disgust

22


23


24


25


26


27


28


29


30


31

Tally your results!

• Should match most of the time

32


33


34


35


36


37


38


39


40


41


42

Ekman - Microexpressions

• Play video• Impact on Darwinian, jamesian, etc?• What is deception w/r emotion?

43

Poker Tells – The Body Language of Poker (1984) Mike Caro

• Poker players exhibit “tells” that reveal valuable information about their hands or intentions to bet

• Some tells are unconscious and directly related to the internal emotion (excitement at a good hand, disappointment at a bad one)

• Some tells are the result of “acting the opposite” to try to deceive the opponents.

• Tells involve facial expressions, gestures, hand positions, body posture, and other behaviors

44

Poker Tells – instant reaction

The Body Language of Poker (1984) Mike Caro45

Poker Tells – acting the opposite

46

Poker Tells – guarding the hand

47

Expressions in virtual 3D faces – FaceGen Neutral Pose

www.facegen.com

48

Expressions in virtual 3D faces – FaceGen Anger

49

Expressions in virtual 3D faces – FaceGen Disgust+smile(open)

50

Expressions in virtual 3D faces – FaceGen blend other expression, phonemes, etc.

51

Next Week

• Hand in either:– A list of possible topics for your course project,

with a paragraph on each, or (if you have already discussed the project with me and decided on one)

– A one page project specification with:• Project goal• What you will do to achieve• What resources you will need• What prior work you can draw on

52

Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1.

Documents

Transcript of Cues to Emotion: Face, Body, Brain Bob Coyne (cs6998) 1.