Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin,...

Visually-induced Visually-induced auditory spatial adaptation auditory spatial adaptation

in monkeys and humansin monkeys and humans

Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh

Center for Cognitive Neuroscience, Duke UniversityHearing Research Center, Boston University

Technical University, Košice, Slovakia

2Nov 6, 2007 SFN 07 San Diego

Introduction

Vision affects auditory spatial perception (e.g. the ventriloquist effect)

Visually-induced shifts in sound localization can persist in the absence of visual stimuli (e.g. barn owl prism adaptation studies)


QUESTION

How does vision calibrate sound perception in primates?

- monkeys and humans

Unlike barn owls, monkeys and humans make eye movements. With every eye movement, the relationship between visual space and auditory space changes.


Visual and auditory spatial information are different!

VISION:

Retina provides “map” of object locations

Locations shift when eyes move

Frame of reference is “eye-centered”


Visual and auditory spatial information are different!

AUDITORY:Sound location calculated from interaural timing and level differences

Cue values do NOT shift when eyes move

Frame of reference is “head-centered”


Goals

Does visual-calibration of auditory space occur in eye-centered, head-centered, or a hybrid coordinate system?

Are humans and monkeys similar?


Experimental Setup

Audiovisual display as viewed by the subject

Horizontal location (degrees)

Ver

tical

loca

tion

(deg

rees

)

9 speakers in front of listener (~1 m distance), separated by 7.5° (humans) or 6° (monkeys)

Light-emitting diodes (LEDs)at three center speakers:- aligned with speakers, or- displaced to the left or to the right (by 5°-humans, 6°-monkeys)

2 LEDs below speaker array used as fixation points (FP)

Stimuli:Auditory stimulus:300-ms broadband noise burstAudio-Visual stimulus:Same noise with synchronously

lid LED.


-40 -30 -20 -10 0 10 20 30 40-40

-30

-20

-10

0

10

20

30

40

stimuli (degree)

resp

onse

(de

gree

)

trained A-only fp

no shift

shift to the same direction as AVshift to the opposite direction as AV

Experiment: Data Presentation Format

Sample Stimulus-Response Raw Data Plot

Actual Stimulus Location (°)

Per

ceiv

ed /

Res

pond

ed L

ocat

ion

(°)

Sample Plot Showing Bias in Responses (Response – Actual Location)

Actual Stimulus Location (°)B

ias

in R

espo

nses

(°) Rightward bias

Leftward bias

Rightw

ard

bias

Leftw

ard

bias


Experiment: Hypothesis

1. Induce Ventriloquist Effect for AV stimuli presented in central sub-region of space, keeping fixation point on the right ( +8°)

2. Test that adaptation results in the effect being present also for Auditory-only stimuli (same FP).

3. Induced shift will decrease outside the trained sub-region (for the same FP).

4. Move FP to the left. Test effect on Auditory-only stimuli:- No change Head-centered.- Response pattern moves with eyes Eye-centered.- Combined representations.

Stimulus Location (°)

Bia

s (°

)

Audiovisual display Expected Responses

FPLEDs

Speakers


Experiment: Procedure

Audiovisual display as viewed by the subject

Horizontal location (degrees)

Ver

tical

loca

tion

(deg

rees

)

One trial consists of:

1. Fixation point (FP) appears.

2. Subject fixates FP.

3. Target stimulus is presented (Audio-Visual or Auditory-only).

4. Subject saccades to perceived location of stimulus (humans instructed to always saccade to sound).

5. Monkeys only: Reward for responding within a criterion window (+- 10° from speaker).


Experiment: ProcedureExperiment divided into 1-hour blocks (12 for humans, 16 for monkeys; 7 humans, 2 monkeys).Within a block three types of trials, randomly interleaved:

Three types of AV stimuli (AV stimulus type kept constant within a block. Data also collected with AV FP on left. These data mirror-flipped to simplify presentation. For monkeys, also AV-aligned stimuli at +-30° to enforce non-linearity):

AV stimuli: 50 %

FPLEDs

Speakers

A-only,trained FP: 25% A-only,shifted FP: 25%

AV stimuli aligned Hypometric shift Hypermetric shift


Results: Humans

Stimulus Location (°)

Audiovisual display Expected Responses

FPLEDs

SpeakersHuman Behavior

Data collapsed acrossdirection of inducedshiftAV responses:- as expected

Trained FP A-onlyresponses:- Shift induced in trained sub-region- Generalization to untrained regions (asymmetrical)

Shifted FP A-onlyresponses:- Shift reduced in center regionHead-centered repre-sentation, modulated by eye position

Mean+SE


Results: Humans vs. MonkeysAudiovisual display

FPLEDs

SpeakersHuman Behavior Monkey data

(only hypometric)

AV responses:- as expected

Trained FP A-onlyresponses:- Shift in trained sub- region weaker- Generalization to untrained regions stronger (asymmetry opposite to humans)

Shifted FP A-onlyresponses:- Shifted with eyes

Representation moreeye-centered

Mean+SE

Monkey Behavior

Mean

6

6-24 0 24

-24 0 24

-24 0 24

Mean+individualsMean+SE

Mean+SE Mean+individuals


SummaryGood news – the main results are consistent across species:

Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region.The pattern of induced shift is modified as the eyes move.

Bad news – there is a lot of differences between species:

Humans MonkeysRepresentation head-centered, eye-modulated eye-centered

Generalizationto untrained sub-regions more on the side away from FP opposite

Difference betweenhyper- and hypometricshifts no yes

Representation whenshift induced on side (data not shown) head-centered, no eye modulation eye-centered


DiscussionThe differences between species can be caused by:

1. differences in neural representation and/or learned behaviorE.g., monkeys often respond by double saccades

2. differences in the stage along the pathway at which the calibration occurs(Insert figure of ear->CN->SOC->IC->MGB->Aud. Ctx->Parietal Ctx->SC->oculomotor)E.g., in humans, the head-centered shift was induced fast, while the eye-centered modulation was much slower different time scales may imply adaptation at different stages

3. procedural differences (e.g., monkeys work for water)

Future work:

Humans: examine temporal and spatial factors influencing the eye-centered modulation.

Monkeys: attempt to induce stronger effect using larger A-V separation.

Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin,...

Documents

Transcript of Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin,...