Do We Click? - Laurent Silbert - H+ Summit @ Harvard
50
Do We Click? Speaker-listener neural coupling underlies successful communication Lauren Silbert 1 , Greg Stephens 2,3 , Uri Hasson 1 1 Princeton Neuroscience Institute and Psychology Department, Princeton University, 2 Lewis-Sigler Institute for Integrative Genomics, and 3 Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ
-
Upload
humanity -
Category
Health & Medicine
-
view
3.152 -
download
1
description
Do we click? Speaker-listener neural coupling underlies successful communication Verbal communication enables us to directly convey information across brains, independent of the actual external state of affairs (e.g. telling a story of past events). Such phenomenon may be reflected in the ability of the speaker to directly induce similar brain patterns in another individual, via speech, in the absence of any other stimulation. The recording of the neural responses from both the speaker brain and the listener brain opens a new window into the neural basis of interpersonal communication, and may be used to assess verbal and non-verbal forms of interaction in both human and other model systems. Further understanding of the neural processes that facilitate neural coupling across interlocutors may shed light on the mechanisms by which our brains interact and bind to form societies. The capacity to communicate internal thoughts from one person to another is at the foundation of human society. Communication naturally requires an interaction between at least two people. Existing neurolinguistic studies are concerned, however, either with speech production or with the comprehension of isolated words or sentences. Little is known, therefore, about the underlying neuronal mechanism that facilitates the transfer of information between two brains during communication. Understanding the interaction between a speaker’s brain and a listener’s brain in the context of real-world communication requires the development of new experimental paradigms. Using function Magnetic Resonance Imaging (fMRI), we measured neural signals from two brains (a speaker and a listener) during a complex everyday communication. We then built a simple, interpretable model that leverages the dynamics of fMRI and uses the speaker’s brain responses as a model for predicting the brain responses within the listener. Our model reveals that during successful communication, the speaker and listener’s brains exhibit joint, temporally coupled, response patterns. Such speaker-listener neural coupling vanishes when participants fail to communicate (such as with different languages). The temporal nature of this speaker-listener coupling suggests that an ability to evoke similar brain patterns in another individual via speech may gate our communication abilities. Moreover, while in most areas the listeners’ brain responses mirror the speaker’s responses with a delay, some areas in the listeners’ brain exhibit predictive anticipatory responses. Finally, we found that the extent of the anticipatory neuronal coupling between interlocutors is predictive of communicative success. Currently a PhD candidate in Neuroscience from Princeton University, Silbert also has a Bachelors from University of Pennsylvania (Biology and Photography), Masters in Neuroscience from Mt. Sinai School of Medicine, and Masters in Psychology from NYU.
Transcript of Do We Click? - Laurent Silbert - H+ Summit @ Harvard
Do We Click? Speaker-listener neural coupling underlies
successful communication
Lauren Silbert1, Greg Stephens2,3, Uri Hasson1
1Princeton Neuroscience Institute and Psychology Department, Princeton University, 2Lewis-Sigler Institute for Integrative Genomics,
and 3 Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ
Verbal communication is a joint activity through which individuals share information
Speaker (production)
Listener (comprehension)
Communication is integral to the functioning of our complex societies
Globalization and the rapidly changing world demand we increase our understanding of how
communication succeeds
Verbal communication is a joint activity through which individuals share information
Speaker (production)
Listener (comprehension)
Speaker (production)
Listener (comprehension)
Neurolinguistic studies constrained by boundaries of individual brains
Speaker (production)
Listener (comprehension)
Neurolinguistic studies constrained by boundaries of individual brains
Control simplified stimuli
Isolation from the environment
Assessing an interactive process: laboratory verse real-world stimuli
red cat bat
Complex natural stimuli
Control simplified stimuli
Isolation from the environment
Two brains interacting
Assessing an interactive process: laboratory verse real-world stimuli
red cat bat
Speaker
Design
15 min monologue
(real life unrehearsed story)
FOMRI™ II Dual Channel MRI Microphone System
Two orthogonal pressure gradient optical microphones
Mic 2: noise
Mic 1: Signal + noise
Extracted signal
Mic 1
Mic 2
Listener Speaker
(n=11)
Design
15 min monologue
(real life unrehearsed story)
(n=11)
Brain activity during production as a model for comprehension
model
(n=11)
Brain activity during production as a model for comprehension
model
(n=11)
Brain activity during production as a model for comprehension
model Listener’s response
Brain activity during production as a model for comprehension
• Bypass the need to specify a priori any formal model of linguistic processes in any given brain area
• Ability to look at similarity of responses during speech production and speech comprehension in the same area
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
(n=11)
Adding Temporal Dynamics to the Model:
1000
Actual correlation coefficient
Conservative Statistics: Parametric and Non-parametric
(n=11)
Correct for multiple comparisons using FDR g=0.05
Analytic F-test applied to overall model fit
Predictions
1. Are neural responses during speech production and speech comprehension coupled?
2. Can the extent of speaker-listener neural coupling predict the success of communication?
Predictions
1. Are neural responses during speech production and speech comprehension coupled?
2. Can the extent of speaker-listener neural coupling predict the success of communication?
subject 4
subject 3
subject 2
subject 1
Brain responses are reliably shared across all listeners during comprehension
f MR
I re
spon
se
time
Speaker’s brain responses during production are coupled to listener’s responses during comprehension
Areas involved in comprehension overlap with areas coupled during communication
I. Is the speaker-listener coupling tied to the content of the story?
• Block Communication • Misaligned Communication
Necessary controls
X
Speaker-Listener neural coupling is absent in the absence of successful communication
Russian Speaker Non-Russian speaking Listeners
?
X
Speaker-Listener neural coupling is absent when communication is incongruous
Speaker telling story 2 Listeners to story 1
?
Necessary controls
I. Is the speaker-listener coupling tied to the content of the story?
• Block Communication • Shuffle Communication
II. Is the speaker a listener of herself? • Temporal dynamics
Brain responses among listeners are time-locked to the moment of vocalization
Average beta weights
Listener’s brain responses mirror the speaker’s brain responses with a delay
Average beta weights
Brain responses shared among listeners are temporally aligned to the moment of vocalization
Temporal dynamics of speaker-listener neural coupling varies across areas
• Dynamics of coupling between a speaker and a listener are fundamentally different from dynamics shared among all listeners.
• Spatial specificity of temporal coupling demonstrates effect cannot be attributed to non-specific, spatially global effects like arousal.
Importance of temporal coupling results:
vs
Predictions
1. Are neural responses during speech production and speech comprehension coupled?
2. Can the extent of speaker-listener neural coupling predict the success of communication?
Level of understanding varies across listeners, as measured by reliable independent raters
The degree of neural coupling predicts the success of communication
Speaker-listener Neural Coupling
Areas where the listener’s responses precede the speaker’s show strongest correlation with behavior
I. In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses.
Summary
I. In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses.
II. The extent of speaker-listener neural coupling is indicative of the success of the communication.
Summary
I. In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses.
II. The extent of speaker-listener neural coupling is indicative of the success of the communication.
III. An ability to evoke similar brain patterns in another individual via speech may gate our communication abilities.
Summary
I. In the course of communication the listener’s brain responses become coupled with the speaker’s brain responses.
II. The extent of speaker-listener neural coupling is indicative of the success of the communication.
III. An ability to evoke similar brain patterns in another individual via speech may gate our communication abilities.
IV. The recording of neural responses from both the speaker brain and the listener brain may be used to assess verbal and non-verbal forms of interaction in both human and other model systems.
Summary
Thank You
Christopher Honey David Heeger Yulia Lerner Bruno Galantucci Chris Thompson Simon Garrod Mina Cikara Alana D'Alfonso
mirror neurons
