Neural Encoding of Speech in Auditory Cortex
Transcript of Neural Encoding of Speech in Auditory Cortex
![Page 1: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/1.jpg)
Neural Encoding of Speech in Auditory Cortex
Jonathan Z. SimonDepartment of BiologyDepartment of Electrical & Computer EngineeringInstitute for Systems Research
University of Maryland
University College London, 22 June 2015http://www.isr.umd.edu/Labs/CSSL/simonlab
![Page 2: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/2.jpg)
AcknowledgementsCurrent (Simon Lab & Affiliates)
Francisco CervantesNatalia LapinskayaMahshid NajafiAlex PresaccoKrishna PuvvadaLisa UiblePeng Zan
Past (Simon Lab & Affiliate Labs)Nayef AhmarSahar AkramMurat AytekinClaudia BoninMaria ChaitMarisel Villafane Delgado Kim DrnecNai Ding Victor Grau-SerratJulian JenkinsDavid KleinLing Ma
Kai Sum LiHuan Luo Raul RodriguezBen WalshJuanjuan XiangJiachen Zhuo
CollaboratorsPamela AbshireSamira AndersonBehtash BabadiCatherine CarrMonita ChatterjeeAlain de CheveignéDidier DepireuxMounya ElhilaliBernhard EnglitzJonathan FritzCindy MossDavid PoeppelShihab Shamma
Funding NIH (NIDCD, NIA, NIBIB); USDA
Past Postdocs & Visitors Aline Gesualdi Manhães Dan HertzYadong Wang
Undergraduate Students Abdulaziz Al-Turki Nicholas AsendorfSonja BohrElizabeth CamengaCorinne CameronJulien DagenaisKatya DombrowskiKevin HoganKevin KahnAlexandria MillerIsidora RanovadovicAndrea ShomeMadeleine VarmerBen Walsh
![Page 3: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/3.jpg)
Outline• Magnetoencephalography (MEG)
• Cortical Representations of Speech
- Encoding vs. Decoding
- Attended vs. Unattended Speech
• Work in Progress
- Attentional Dynamics
- Aging and the Cocktail Party Problem
- Foreground vs. Background
![Page 4: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/4.jpg)
Magnetoencephalography• Non-invasive, Passive, Silent
Neural Recordings
• Simultaneous Whole-Head Recording (~200 sensors)
• Sensitivity• high: ~100 fT (10–13 Tesla)• low: ~104 – ~106 neurons
• Temporal Resolution: ~1 ms
• Spatial Resolution• coarse: ~1 cm• ambiguous
![Page 5: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/5.jpg)
Neural Signals & MEG
tissue
CSF
skull
scalpB
MEG
VEEG
recordingsurface
currentflow
orientationof magneticfield
MagneticDipolarField
Projection
•Direct electrophysiological measurement•not hemodynamic•real-time
•No unique solution for distributed source
Photo by Fritz Goro
•Measures spatially synchronized cortical activity
•Fine temporal resolution (~ 1 ms)•Moderate spatial resolution (~ 1 cm)
![Page 6: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/6.jpg)
Time Course of MEG Responses
BroadbandNoise
Auditory Evoked Responses
• MEG Response Patterns Time-Locked to Stimulus Events
• Robust
• Strongly Lateralized
Pure Tone
![Page 7: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/7.jpg)
Component Analysis• Each component has both
spatial and temporal profile
• Data driven, e.g., PCA, ICA, DSS
• DSS: ordered by trial-to-trial reproducibility
• ➔ Spatial Filter, e.g. for single trials
• Can analyze temporal processing separately from anatomical origin Särelä & Valpola (2005)
de Cheveigné & Simon, J. Neurosci. Methods (2008)
![Page 8: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/8.jpg)
MEG Responses
AuditoryModel
to Speech Modulations
![Page 9: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/9.jpg)
Ding & Simon, J Neurophysiol (2012) “Spectro-Temporal Response Function”
(up to ~10 Hz)
MEG Responses Predicted by STRF Model
Linear Kernel = STRF
![Page 10: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/10.jpg)
Ding & Simon, J Neurophysiol (2012)Zion-Golumbic et al., Neuron (2013)
Neural Reconstruction of Speech Envelope
2 s
stimulus speech envelopereconstructed stimulus speech envelope
Reconstruction accuracy comparable to single unit & ECoG recordings
(up to ~ 10 Hz)
MEG Responses
...
DecoderSpeech Envelope
![Page 11: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/11.jpg)
Ding & Simon, J Neurophysiol (2012)Zion-Golumbic et al., Neuron (2013)
Neural Reconstruction of Speech Envelope
2 s
stimulus speech envelopereconstructed stimulus speech envelope
Reconstruction accuracy comparable to single unit & ECoG recordings
(up to ~ 10 Hz)
MEG Responses
...
DecoderSpeech Envelope
![Page 12: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/12.jpg)
Ding & Simon, J Neurophysiol (2012)Zion-Golumbic et al., Neuron (2013)
Neural Reconstruction of Speech Envelope
2 s
stimulus speech envelopereconstructed stimulus speech envelope
Reconstruction accuracy comparable to single unit & ECoG recordings
(up to ~ 10 Hz)
MEG Responses
...
DecoderSpeech Envelope
![Page 13: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/13.jpg)
Neural Representation of Speech: Temporal
![Page 14: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/14.jpg)
Speech in Noise
Ding & Simon, J Neuroscience (2013)
![Page 15: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/15.jpg)
Speech in Noise
Ding & Simon, J Neuroscience (2013)
![Page 16: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/16.jpg)
Speech in Noise: Results
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cy
Ding & Simon, J Neuroscience (2013)
![Page 17: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/17.jpg)
Speech in Noise: Results
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cy
Ding & Simon, J Neuroscience (2013)
![Page 18: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/18.jpg)
Speech in Noise: Results
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cy
Ding & Simon, J Neuroscience (2013)
![Page 19: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/19.jpg)
Speech in Noise: Results
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cy
Ding & Simon, J Neuroscience (2013)
![Page 20: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/20.jpg)
Speech in Noise: Results
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cy
+6 dB
-6 dB1 s
A Neural Reconstruction ofUnderlying Speech Envelope
B Reconstruction Accuracy
corre
latio
n
SNR (dB)Q +6 +2 −3 −6 −9
0
.1
.2
0 25 50 75 1000
.1
.2
.3
C Correlation with Intelligiblity
intelligiblity (%)
reco
nstru
ctio
n ac
cura
cyacross Subjects
Ding & Simon, J Neuroscience (2013)
![Page 21: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/21.jpg)
Noise-Vocoded Speech
Ding, Chatterjee & Simon, NeuroImage (2014)
“in noise” = +3 dB SNR
![Page 22: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/22.jpg)
Noise-Vocoded Speech: Results
• Cortical entrainment to natural speech robust to noise• Cortical entrainment to vocoded speech is not• Not explainable by passive envelope tracking mechanisms
- noise vocoding does not directly affect the stimulus envelope
![Page 23: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/23.jpg)
Noise-Vocoded Speech: Results
![Page 24: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/24.jpg)
Cortical Speech Representations
• Neural Representations: Encoding & Decoding
• Linear models: Useful & Robust
• Speech Envelope only (as seen by MEG)
• Envelope Rates: ~ 1 - 10 Hz
![Page 25: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/25.jpg)
Alex Katz, The Cocktail Party
The Cocktail Party
![Page 26: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/26.jpg)
Alex Katz, The Cocktail Party
The Cocktail Party
![Page 27: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/27.jpg)
Alex Katz, The Cocktail Party
The Cocktail Party
![Page 28: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/28.jpg)
Alex Katz, The Cocktail Party
The Cocktail Party
![Page 29: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/29.jpg)
Alex Katz, The Cocktail Party
The Cocktail Party
![Page 30: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/30.jpg)
speech
competing speech
Experiments
![Page 31: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/31.jpg)
speech
competing speech
Experiments
![Page 32: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/32.jpg)
reverberation
Experiments in Progress
![Page 33: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/33.jpg)
speech
competing speech
Experiments in Progress
olderlistener
![Page 34: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/34.jpg)
speech
competing speech
Experiments in Progress
competing speech
![Page 35: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/35.jpg)
speech
competing speech
Two Competing Speakers
![Page 36: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/36.jpg)
Selective Neural Encoding
![Page 37: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/37.jpg)
Selective Neural Encoding
![Page 38: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/38.jpg)
Selective Neural Encoding
![Page 39: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/39.jpg)
Unselective vs. Selective Neural Encoding
![Page 40: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/40.jpg)
Unselective vs. Selective Neural Encoding
![Page 41: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/41.jpg)
Selective Neural Encoding
![Page 42: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/42.jpg)
Stream-Specific Representation
grand average over subjects
representative subject
Identical Stimuli!
reconstructed from MEG
attended speech envelopes
reconstructed from MEG
attending tospeaker 1
attending tospeaker 2
Ding & Simon, PNAS (2012)
![Page 43: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/43.jpg)
Stream-Specific Representation
grand average over subjects
representative subject
Identical Stimuli!
reconstructed from MEG
attended speech envelopes
reconstructed from MEG
attending tospeaker 1
attending tospeaker 2
Ding & Simon, PNAS (2012)
![Page 44: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/44.jpg)
Single Trial Speech Reconstruction
Ding & Simon, PNAS (2012)
![Page 45: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/45.jpg)
Single Trial Speech Reconstruction
Ding & Simon, PNAS (2013)
![Page 46: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/46.jpg)
Overall Speech Reconstruction
0.2
0
0.1
corre
latio
n
attended speechreconstruction
backgroundreconstruction
attended speech background
Distinct neural representations for different speech streams
![Page 47: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/47.jpg)
Invariance Under Relative Loudness Change?
![Page 48: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/48.jpg)
Invariance Under Relative Loudness Change?
![Page 49: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/49.jpg)
Invariance under Relative Loudness Change
attended
backgroundcorr
elat
ion
.1
.2
-8 -5 0 5 8Speaker Relative Intensity (dB)
Neural Results
• Neural representation invariant to relative loudness change
• Stream-based Gain Control, not stimulus-based
![Page 50: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/50.jpg)
Forward STRF Model
Spectro-Temporal Response Function (STRF)
![Page 51: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/51.jpg)
Forward STRF Model
Spectro-Temporal Response Function (STRF)
![Page 52: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/52.jpg)
STRF Results
•STRF separable (time, frequency)•300 Hz - 2 kHz dominant carriers•M50STRF positive peak•M100STRF negative peak
TRF
•M100STRF strongly modulated by attention, but not M50STRF
attended
.2
.5
1
3
0 100 200
Background
fre
qu
en
cy (
kH
z)
.2
.5
1
3
0 100 200
Attended
time (ms) time (ms)
background
![Page 53: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/53.jpg)
STRF Results
•STRF separable (time, frequency)•300 Hz - 2 kHz dominant carriers•M50STRF positive peak•M100STRF negative peak
TRF
•M100STRF strongly modulated by attention, but not M50STRF
attended
.2
.5
1
3
0 100 200
Background
fre
qu
en
cy (
kH
z)
.2
.5
1
3
0 100 200
Attended
time (ms) time (ms)
background
![Page 54: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/54.jpg)
STRF Results
•STRF separable (time, frequency)•300 Hz - 2 kHz dominant carriers•M50STRF positive peak•M100STRF negative peak
TRF
•M100STRF strongly modulated by attention, but not M50STRF
attended
.2
.5
1
3
0 100 200
Background
fre
qu
en
cy (
kH
z)
.2
.5
1
3
0 100 200
Attended
time (ms) time (ms)
background
![Page 55: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/55.jpg)
Neural Sources
RightLeft
anterior
posterior
medial
M50STRFM100STRFM100
•M100STRF source near (same as?) M100 source: Planum Temporale
•M50STRF source is anterior and medial to M100 (same as M50?): Heschl’s Gyrus
5 mm
•PT strongly modulated by attention, but not HG
![Page 56: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/56.jpg)
Cortical Object-Processing Hierarchy
0 100 200 400time (ms)
0
attendedbackground
Attentional Modulation
0 100 200 400
0
time (ms)
clean
-5 dB-8 dB
Influence of Relative Intensity
0 dB5 dB8 dB
•M100STRF strongly modulated by attention, but not M50STRF.•M100STRF invariant against acoustic changes.•Objects well-neurally represented at 100 ms, but not 50 ms.
![Page 57: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/57.jpg)
Studies In Progress
• Attentional Dynamics
• Aging & Neural Representations of Speech
• Neural Representations of the Background
![Page 58: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/58.jpg)
Attentional DynamicsAttend to Speaker 1
Attend to Speaker 2
Prob
abilit
y of
atte
ndin
g S
peak
er 1
![Page 59: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/59.jpg)
Attentional DynamicsAttend to Speaker 1
Switch Attention
Attend to Speaker 2
Prob
abilit
y of
atte
ndin
g S
peak
er 1
![Page 60: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/60.jpg)
Younger vs. Older Listeners
withCompetingSpeaker
Spee
ch R
econ
stru
ctio
n
withCompetingSpeaker
Older AdultsYounger Adults
In Quiet
Integration window (ms)
In Quiet
![Page 61: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/61.jpg)
Younger vs. Older Listeners
withCompetingSpeaker
Spee
ch R
econ
stru
ctio
n
withCompetingSpeaker
Older AdultsYounger Adults
In Quiet
Integration window (ms)
In Quiet
![Page 62: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/62.jpg)
speech
competing speech
Three Competing Speakers
competing speech
![Page 63: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/63.jpg)
Foreground vs. Background
![Page 64: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/64.jpg)
Foreground vs. Background
![Page 65: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/65.jpg)
Foreground vs. Background
![Page 66: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/66.jpg)
Foreground vs. Background
![Page 67: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/67.jpg)
Foreground vs. Background
![Page 68: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/68.jpg)
Foreground vs. Background
![Page 69: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/69.jpg)
Foreground vs. Background
![Page 70: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/70.jpg)
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
Backgrounds vs. Background
NoiseFloor
ForegroundIndividualBackgrounds
FusedBackground
Individual Speech Streams
Integration Window over Late Times Only
![Page 71: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/71.jpg)
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
Backgrounds vs. Background
NoiseFloor
IndividualBackgrounds
FusedBackground
Foreground
Individual Speech Streams
Integration Window over Late Times Only
![Page 72: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/72.jpg)
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
Backgrounds vs. Background
NoiseFloor
IndividualBackgrounds
FusedBackground
Foreground
Individual Speech Streams
Integration Window over Late Times Only
![Page 73: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/73.jpg)
Backgrounds vs. BackgroundWhy not?
Speaker 1
Two Speakers
Speaker 2
Stimulus Background
MEG Response
![Page 74: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/74.jpg)
Backgrounds vs. BackgroundWhy not?
Speaker 1
Two Speakers
Speaker 2
Stimulus Background
MEG Response
![Page 75: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/75.jpg)
Backgrounds vs. BackgroundWhy not?
Speaker 1
Two Speakers
Speaker 2
Stimulus Background
MEG Response
![Page 76: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/76.jpg)
Backgrounds vs. BackgroundWhy not?
Speaker 1
Two Speakers
Speaker 2
Stimulus Background
MEG Response
![Page 77: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/77.jpg)
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
Backgrounds vs. Background
NoiseFloor
IndividualBackgrounds
FusedBackground
Foreground
Individual Speech Streams
Integration Window over Late Times Only
![Page 78: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/78.jpg)
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
Backgrounds vs. Background
NoiseFloor
IndividualBackgrounds
FusedBackground
Foreground
Individual Speech Streams
Integration Window over Late Times Only
![Page 79: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/79.jpg)
Backgrounds vs. Background
−0.2
0
0.2
0.4
Noise FloorSummed Individual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams II
NoiseFloor
IndividualBackgrounds
Summed
FusedBackground
Foreground
Individual Speech Streams
Integration Window over Late Times Only
![Page 80: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/80.jpg)
Backgrounds vs. Background
−0.2
0
0.2
0.4
Noise FloorSummed Individual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams II
NoiseFloor
IndividualBackgrounds
Summed
FusedBackground
Foreground
Individual Speech Streams
+
+
Integration Window over Late Times Only
![Page 81: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/81.jpg)
Backgrounds vs. Background
−0.2
0
0.2
0.4
Noise FloorSummed Individual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams II
NoiseFloor
IndividualBackgrounds
Summed
FusedBackground
Foreground
Individual Speech Streams
+
+
Integration Window over Late Times Only
![Page 82: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/82.jpg)
Backgrounds vs. Background
−0.2
0
0.2
0.4
Noise FloorSummed Individual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams II
NoiseFloor
IndividualBackgrounds
Summed
FusedBackground
Foreground
Individual Speech Streams
?
+
+
Integration Window over Late Times Only
![Page 83: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/83.jpg)
Backgrounds vs. Background
−0.2
0
0.2
0.4
Noise FloorSummed Individual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams II
NoiseFloor
IndividualBackgrounds
Summed
FusedBackground
Foreground
Individual Speech Streams
?
+
+
Integration Window over Late Times Only
![Page 84: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/84.jpg)
Backgrounds vs. Background
Individual Backgrounds Summed
Fuse
d Ba
ckgr
ound
0 0.1 0.2 0.3
0
0.1
0.2
0.3
Background Representations
individual backgrounds reconstruction
fuse
d ba
ckgr
ound
reco
nstru
ctio
n
+
Integration Window over Late Times Only
![Page 85: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/85.jpg)
Backgrounds vs. Background
Individual Backgrounds Summed
Fuse
d Ba
ckgr
ound
0 0.1 0.2 0.3
0
0.1
0.2
0.3
Background Representations
individual backgrounds reconstruction
fuse
d ba
ckgr
ound
reco
nstru
ctio
n
+
Integration Window over Late Times Only
![Page 86: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/86.jpg)
Backgrounds vs. Background
Individual Backgrounds Summed
Fuse
d Ba
ckgr
ound
0 0.1 0.2 0.3
0
0.1
0.2
0.3
Background Representations
individual backgrounds reconstruction
fuse
d ba
ckgr
ound
reco
nstru
ctio
n
+
High latency areas (PT) represent fused background with better fidelity than individual backgrounds(p = 0.012)
Integration Window over Late Times Only
![Page 87: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/87.jpg)
Foreground vs. Background
−0.2
0
0.2
0.4
Noise FloorIndividual BackgroundsFused BackgroundForeground
reco
nstru
ctio
n ac
cura
cy
Individual Speech Streams
ForegroundIndividualBackgrounds
Individual Speech Streams
Integration Window over Late Times Only
![Page 88: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/88.jpg)
Foreground vs. BackgroundEarly vs. Late
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Core Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Higher Order Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
p > 0.10 p < 1e-11
Early (HG) Late (PT)
Background Background
Fore
grou
nd
![Page 89: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/89.jpg)
Foreground vs. BackgroundEarly vs. Late
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Core Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Higher Order Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
p > 0.10 p < 1e-11
Early (HG) Late (PT)
Background Background
Fore
grou
nd
![Page 90: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/90.jpg)
Foreground vs. BackgroundEarly vs. Late
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Core Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
0 0.1 0.2 0.3−0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35Higher Order Representations
backgrounds reconstruction
fore
grou
nd re
cons
truct
ion
p > 0.10 p < 1e-11
Early (HG) Late (PT)
HG represents attended and unattended speech with almost equal fidelity
Background Background
Fore
grou
nd
![Page 91: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/91.jpg)
Summary• Cortical representations of speech- representation of envelope (up to ~10 Hz)
• Cortical Processing Hierarchy: Consistent with being neural representation of auditory perceptual object
• Object representation at 100 ms latency (PT), but not by 50 ms (HG)
• Preliminary evidence for - PT: additional fused background representation
- HG: almost equal representations
![Page 92: Neural Encoding of Speech in Auditory Cortex](https://reader033.fdocuments.us/reader033/viewer/2022042403/625d8e94dfbf6e3e047b2a59/html5/thumbnails/92.jpg)
Thank You