Facial identity and expression perception in the human visual system

Human Vision and Eye Movement Laboratory

Medicine (Neurology)

Ophthalmology and visual sciences

Psychology

University of British Columbia

NANOS 2011 Vancouver

DISCLOSURES:

NO COMMERCIAL/PROPRIETARY INTERESTS

FUNDING SUPPORT:

NIMH 1R01 MH069898

CIHR MOP-77615, MOP-85004, MOP-102567, MOP-106511

NSERC RGPIN 355879-08

Michael Smith Foundation for Health Research CI-SSH-035(03-1)

Canada Research Chairs 950-202111

Student support from MSFHR, CIHR, Alzheimer’s Foundation

Canada Research Chairs

Bruce and Young (1986) cognitive model

COMMON CONCEPT: perception of identity and expression emerge from divergent streams of face processing

EXPRESSION

Encode facial dynamics

Invariant for static structure

• allow generalization across people

IDENTITY

Encode static structure

Invariant for dynamic changes

•allow stable recognition despite changes in expression and age

COMMON CONCEPT: perception of identity and expression emerge from divergent streams of face processing

• Are expression and identity functionally and structurally independent?

If so, this would imply that the human brain has evolved highly specific systems for extracting different structural properties of the same object.

• 1. Behavioural studies

• 2. Anatomic fMRI studies

• 3. Patient studies

1. BEHAVIOURAL INVESTIGATION

use FACE-ADAPTATION to probe the functional relationship between the perception of facial expression and perception of facial identity

1. Behavioural adaptation

90/10 70/30 50/50 30/70 10/90

fear anger

a. Expression adaptation: are these invariant for the identity of the face?

Method - PERCEPTUAL BIAS TECHNIQUE- create a morph continuum between two

expressions:

- stare at one ‘end-face’ for 5 sec.. ..then flash a 300ms ambiguous morph face..

..and ask a question:

“Did you see fear or anger?”

“EXPRESSION AFTER-EFFECT”: Subjects are more likely to perceive the other expression

ambiguous stimuli

1. Behavioural adaptation

5-second Adapting stimulus

300 ms Probe stimulus

(*three expression pairs: Angry/afraid; happy/sad; disgusted/surprised)

Same person

Different person,Same gender

Different person,Different gender

• Changing identity reduces but does not eliminate the expression after-effect.

• Expression representations may have two components:

1. Behavioural adaptation

}identity-dependent component

}

identity-invariant component

Afte

reffe

ct m

agn

itud

e (%

)

90/10 70/30 50/50 30/70 10/90

“Bob” “Joe”

b. Identity adaptation: is it invariant for the expression of the face?

Method - create a morph continuum between two different people’s faces:

- after adaptation to BOB, subjects are more likely to respond that a following ambiguous face is more like JOE.

1. Behavioural adaptation

Is the identity aftereffect partially dependent on expression?

1. Behavioural adaptation

Same expressionDifferent expression

Answer: NO. In contrast to expression aftereffects, there is no expression-dependent component of identity aftereffects.

Afte

reffe

ct m

agn

itud

e

Would we find expression-dependency in the identity after-effect for more familiar faces?

notfamiliar

minimallyfamiliar

culturallyfamiliar

veryfamiliar

Answer: NO

1. Behavioural adaptation

Afte

reffe

ct m

agn

itud

e

Both contribute to expression adaptation

identity adaptation

There is an interesting asymmetric relationship between identity and expression!

One possible (speculative) interpretation, influenced by network models….

1. Behavioural adaptation

1. STRUCTURAL INVESTIGATION

Use fMRI-ADAPTATION to probe the structural relationship between perception of facial expression vs. identity

Fusiform face area Superior temporal sulcus

Expression morphingIdentity m

orphing2-D Morph Matrix with “categorical boundaries”:

EX

PR

ES

SIO

N B

OU

ND

AR

Y

2. fMRI adaptation

• Ultimately, what mattered most in this experiment was not the stimulus but the perceptual decision - whether the subject said that they saw the face as similar or different.

IDENTITY BOUNDARY

2. Rapid event-related fMRI adaptation study

identical establishes baseline of full adaptation

In all of these, face 2 is 33% morph different from face 1.

Subject sees pairs of faces:

Do FFA and STS differ in adaptation effects induced by:

• physical changes in expression vs identity?

• perception of difference in expression vs identity?

Subjects perform 2 runs with the same stimulus set:

•Expression task

•Identity task

similar expression

different expression

identity is identical (held constant):

similar identity

different identity

expression is identical (held constant):

same samediff diffidentity expression

FUSIFORM FACE AREA (FFA)

* FFA is sensitive to any structural change

2. fMRI adaptation

Baseline = adaptation for repetition of identical face

Release from adaptation (area ‘detects’ a difference):

# FFA adaptation release is more when subject perceives a difference in EITHER identity or expression

(as reported by subject)

SUPERIOR TEMPORAL SULCUS (STS)

identity task

expression task

samediff diffidentity expression

same samediff diffidentity expression

same

• STS is modulated by TASK - differences only seen when subject is actively processing expression

2. fMRI adaptation

DURING EXPRESSION TASK:

* STS is sensitive to any stimulus change,

# STS adaptation release is more when subject perceives a difference in EITHER identity or expression

Conclusions of fMRI experiment

• FFA and STS activity is sensitive to physical properties of face,

• Correlate with perceptual report of subject, for both identity AND expression

• STS is recruited during expression task

2. fMRI adaptation

• Identity and expression specific signals not seen until later in cortical hierarchy:

Identity – precuneus Expression – middle STS

1. NEUROPSYCHOLOGY

Differential impact of LESIONS on the perception of facial expression and perception of facial identity

3. Lesion studyRegion of interest

Subject

prosopagnosic

prosopagnosic

prosopagnosic

fMRI localization data in patients

expression

ide

ntit

y

Using 2-D morph matrix, create 4 oddity paradigm tests:

Which face is the different expression?

Identity varies

Expression varies Expression fixed

Identity fixed

Which face is the different person?

We vary the level of difficulty by choosing morph faces of varying distance in the matrix

3. Lesion study: are expression and identity differentially impaired?

3. Lesion study

Identity task Expression task

expression varying

expression fixed

identityvarying

identity fixed

• Identity but not expression impaired by lesions of:

OFA/FFAAnt temporal lobeMedial fusiform

• Expression but not identity impaired by lesion of:

STS

• Expression constancy of identity judgments impaired by STS lesion!

Other dimension fixed

Impairedexpression only

Impairedexpression ANDImpaired identity

ImpairedIdentity only

normal

Summary

Behavioural adaptation results: • identity representations are expression-invariant• expression representations have identity-dependent and identity-invariant components

fMRI-adaptation results:• effects of perceptual decision for both expression and identity are REFLECTED in the activity of both FFA and pSTS (does NOT mean that they are encoded there).

Lesion results:• “fusiform - anterior temporal stream” lesions impair identity perception but NOT expression perception• STS lesion impaired expression perception and impedes identity judgments when expression must be discounted.

1. Behavioural adaptation

Middle STS

Posterior STS

Insula Amygdala

FFA

Anteriortemporal

Precuneus?

Other fusiform regions

Bicycle tour of Vancouver! WEDNESDAYmeet at registration desk 1200h

Behavioural adaptation: • Fox CJ, Barton JJS. What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res 2007; 1127: 80-9.• Butler A, Oruc I, Fox CJ, Barton JJS. Factors contributing to the adaptation aftereffects of facial expression. Brain Res 2008; 1191: 116-26.• Fox CJ, Oruc I, Barton JJS. It doesn’t matter how you feel. The facial identity aftereffect is invariant to changes in facial expression. J Vision 2008; 8(3): 11.1-13.fMRI-adaptation:• Fox CJ, Moon S-Y, Iaria G, Barton JJS. The correlates of subjective perception of identity and expression in the face network: an fMRI adaptation study. Neuroimage 2009; 44: 569-80.Lesions:• Fox CJ, Iaria G, Duchaine BC, Barton JJS. Behavioral and fMRI studies of identity and expression perception in acquired prosopagnosia. Vision Sciences Society, Naples 2008

PERSONNELAndrea Butler Brad Duchaine Chris Fox Giuseppe IariaSo Young Moon Ipek Oruç

3. Lesion study

Desired test properties:

• equivalent perceptual difficulty, • controls perform well but not at ceiling• low variance

CONTROL DATA: