SOME ARTICULATORY AND CEREBRAL CORRELATES OF · PDF fileSOME ARTICULATORY AND CEREBRAL...
Transcript of SOME ARTICULATORY AND CEREBRAL CORRELATES OF · PDF fileSOME ARTICULATORY AND CEREBRAL...
SOME ARTICULATORY AND CEREBRAL CORRELATES
OF PROSODIC FOCUS IN FRENCH
Hélène Lœvenbruck
Speech and Cognition Department, GIPSA-lab (ex ICP)
Grenoble, France
GrenobleImagesParoleSignal
Automatique
� Prosodic focus
- draws the listener/viewer’s attention to relevant elements in the discourse.
- a means to designate, identify or select an element
(e.g. Halliday, 1967; Gussenhoven, 1983; Selkirk 1984; Nølke, 1994; Birch & Clifton, 1995; Ladd, 1996)
=> Related to pointing, or deixis
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
What’s pointing?
- a communicative device which orients the attention of another person so that an object / person / direction / event becomes the shared focus of attention
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
- performed with the index finger and arm extended in the direction of the interesting object and withthe other fingers curledinside the hand
(Butterworth, 2003)
Pointing is ‘special’ because it is…
…ubiquitous
- observed in everyday interactions (work, vacation), in most cultures of the world
- observed in oral as well as signed communication- used linguistically for referential indexing (pronouns) in some sign languages (Klima & Bellugi, 1988)
…probably universal
- index finger pointing claimed to be a universal ability (e.g. Povinelli & Davis, 1994)-- in some cultures, index finger pointing can be replaced with lip pointing (Barai in Papua New Guinea, see Wilkins, 2003)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Pointing is ‘special’ because it…
…has a long historical past
- European art:
Leonardo da Vinci (late XVth century) Jean Fouquet (early XVth century)Bayeux Tapestry (end of the XIth century)
- Asian art:
Sculptures of pointing buddhas ( XVIIIth century Burmese art)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Pointing is ‘special’ because it…
…is performed by evolved animals, capable of some degree of imitation and inter-individual communication
- Chimpanzees in captivity can point at unreachable food.
- They sometimes use ‘canonical’ index finger pointing, but most often, they point with all fingers extended.
- They do so without explicit training.
(Leavens, Hopkins & Bard, 2005)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Pointing is ‘special’ because it is… multimodal
� digital
� ocular
� vocal
� labial (Kuna indians in Panama, Arrernte in Australia, Awtuwand Barai in Papua New Guinea, Ewe in Ghana, Navajo in North America; cf. Wilkins, 2003)
� with other body parts(head, lip, chin, elbow, foot, arm, hand: Kendon, 1996)
ThatThat’’ss Andy!Andy!
Pointing is ‘special’ because it is…
…involved in stages of language development
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
- Deictic gaze (8-9 months)
mutual attention between child and adult(e.g. Tomasello, 1999)
- Finger pointing (9-11 months)
lexicon construction: « what’s that? »(e.g. Butterworth & Morissette , 1996)
- Word + pointing gesture (16-20 months)
morphosyntax emergence
(Volterra & Caselli, 1986; Goldin-Meadow, 1999; Bates & Dick, 2002; Butterworth, 1997)
Pointing is involved in stages of language development
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Demonstratives in acquisition:
Syntactic precursors
- « That! » presentational
- « That’s Mum! » presentational construction
- « That’s the sugar that goes in there! » relative construction
- « That’s Mum that said that the sugar goes in there! »
(Jisa & Kern, 1998; Diessel & Tomasello, 2000; Tomasello, 2003)
� Pointing (ocular, manual) is involved at many stages
of human language development, from mutual attention
emergence (gaze) to syntactic acquisition through
lexicon construction.
� What do we know about the somatosensory and
cerebral correlates of vocal pointing?
� The role of (ocular and manual) pointing in language
development suggests that vocal pointing may share
features in common with ocular and manual pointing,
and typically may be grounded in close cerebral tissues.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Could vocal pointing share cerebral features in common
with ocular and manual pointing?
What are the cerebral correlates
of manual and ocular pointing?
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Manual pointing:
-Preparation phase: prepare to point with right index finger to an cued location (left or right)
- Execution phase: when target flashes, point towards target
Ocular pointing (saccades):
- Preparation phase: prepare to look at cued location (left or right)
- Execution phase: when target flashes, look at target location
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
- Bilateral occipital cortex: transient sensory response to cue
- Bilateral frontal cortex: sustained responseat the junction of precentral and superior frontal sulci, i.e. human frontal eye field (FEF, involved in oculomotorpreparation)
- Bilateral posterior parietal cortex: sustained response along horiz. segment of intraparietal sulcus (IPS)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
- Bilateral FEF: same as in ocular pointing
- Bilateral IPS: same as in ocular pointing
- Additional left H. activation for manual pointing:angular gyrus, supramarginal g., superior parietal lobule (PrCu), dorsal precentral g., superior temporal sulcus
Results hold, even when left hand is used
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
What are the cerebral correlates
of vocal pointing?
In French, vocal pointing can be conveyed by:
- syntactic extraction:
C’est Madeleine qui m’amena
(It’s/that’s Madeleine who/that brought me along)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
- Prosodic focus:
MADELEINEFm’amena
(MADELEINEF brought me along)
MADELEINE m’amena
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� Acoustically sophisticated:
Prosodic focus = F0 raising + lengthening on focal constituent
F0 lowering on pre-focal constituent
Deaccentuation of post-focal constituent
Rhythmical effects, e.g. increased duration of
focal constituent
See e.g. Astésano (2001), Dahan & Bernard (1996), Delais-Roussarie et al. (2002),
Di Cristo (1998), Dohen & Lœvenbruck (2004), Jun & Fougeron (2000), Rossi
(1999), Touati (1987)
MADELEINE m’amena
RA NI MA
� Rythmically:
- Increase in duration of focal constituent
- Increase in duration of focused syllable (onset)
- Sometimes anticipatory increase in duration of pre-focal
syllable
- No decrease in duration of post-focal constituent (no
dephrasing)
� Prosodic structure:
- Sometimes restructuring of focused AP into one IP or into
several APs.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� Are there articulatory correlates of prosodic pointing?
DiscussionMultimodal pointing
-acoustic-articulatory-cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
(old) EMA acquisition protocole
5 coils in sagittal plane:
– 1 reference (sup. incisor)
– 3 on the tongue
– 1 for the jaw (lower incisor)
Fe = 500 Hz
DiscussionMultimodal pointing
-acoustic-articulatory-cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Position of Ymid(mm)
Velocity(mm/s)
Acceleration
(mm/s2)
Beg. /la/
Peak velocity
Max displact
end /la/
DiscussionMultimodal pointing
-acoustic-articulatory-cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Displacement in [la]
0
2
4
6
8
10
12
utt. initial utt. central utt. final
mm unfocused
focused
Duration of [la]
0
50
100
150
200
250
utt. initial utt. central utt. final
ms unfocused
focused
Lœvenbruck (1999)
Another articulatory monitoring tool: Liptrack
Lipopening(/a/)
Lip closure(/m/)
LipProtrusion
Inter-lip areaDiscussionMultimodal
pointing-acoustic-articulatory-cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� Articulatorily, prosodic pointing requires accurate motor control:
-increased articulation and duration of the focal constituent
-with optional reduced articulation of post-focal constituent
Dohen et al. (2004); Dohen & Lœvenbruck (2006)
unfocused /ma/
focused /ma/
post-focal /ma/
DiscussionMultimodal pointing
-acoustic-articulatory-cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
What are the cerebral correlates
of vocal pointing (deixis)?
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
16 right-handed male French speakers
(1) Baseline condition
Madeleine m’amena /ma.d�.l�n.ma.m�.na/
(Madeleine brought me around)
(2) Prosodic pointing condition
MADELEINEF m’amena
(3) Syntactic pointing condition
C’est Mad’leine qui m’am’na /s�.ma.dl�n.ki.ma.mna/
(It’s Mad’leine who brought me ’round)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Madeleine m’amena
3s +
3s x
3s
MADELEINE m’amena
3s
…14 silent productions
of the sentence = 42s
� Audio validation
Duration and F0 analyses using Praat software.
Checked that pre- and post-scan audio recordings were similar.
� Paradigm and task:
-extensive training (inner speech) a day before the experiment
-pre- and post-scan audio recordings
to assess subject’s performance inside scanner
-3 functional scans using a block paradigm
-each scan = 8 epochs (each condition twice) of 42 s
-order of presentation alternated between scans and subjects
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
�MRI acquisition
-1.5 MR imager (philips NT) with EPI
-25 adjacent, axial, slices (5mm) were imaged 10 times during each
epoch
-imaging volume parallel to the bi-commissural plane
-major MR parameters:
TR = 3700 ms
TE = 45 ms
pulse angle = 90°
acquisition matrix = 64x64
reconstruction matrix = 128x128
field-of-view = 256x256 mm2
-high-resolution 3D anatomical MR scan between first and second
functional scans.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� MRI data processing
SPM-99 software
Pre-processing steps:
- Motion correction
- Spatial normalization with MNI template
- Spatial smoothing
� Statistics
Contrasts between conditions using the GL Model.
Statistical significance threshold p = 0.001.
Clusters of activated pixels were identified, based on the intensity of
the individual responses and the spatial extent of the clusters.
Finally, a significance threshold of p = 0.05 (corrected) was applied
for identification of the activated clusters.
Fixed effect group analysis.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
LIFG + insula
Premotorcortex
Wernicke
SMG
sensory
expectation
auditory
descriptionaction analysis
RR LL
Prosodic pointing recruits a left temporo-parieto-frontal network
Cf. TPF in action imitation (Iacoboni et al., 2003)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� The description of the actions to be imitated is handled by the
superior temporal sulcus (STS) and sent to the posterior parietal
cortex, where it is combined with additional somatosensory
information.� This completed description is then sent to the frontal cortex where
the goal of the actions to be imitated is coded.
� Reafferent copies of the imitated actions are sent back to the STS
for action monitoring.
Temporo-Parieto-Frontal network
in vocal pointing
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
� A left temporo-parieto-frontal network might be recruited in the
organization of vocal pointing actions from auditory representations.
� The inferior parietal region seems to function as a sensory
integrator forming representations necessary in the organization of
linguistic pointing at targets.
� In analogy with visually-guided manual pointing, prosodic pointing
may need integrated representations (auditory and articulatory) to
be formed via the superior temporal and parietal regions in order to
organize articulation and phonation in an adequate prosodic pattern.
� Prosodic pointing, i.e. expressive orofacial pointing may be
considered in continuity with ocular and manual pointing.
…To
From…
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Pointing with syntax
Premotorcortex
LIFG + insula
� Syntactic (with or without prosody) pointing is grammaticalised,
automatized.
It does not recruit the entire TPF network and is handled solely by
the left IFG.
� No activation of the LSMG or Wernicke’s area (no sensory or
auditory expectations)
X
LLRR
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
What are the cerebral correlates
of multimodal pointing?
Multimodal Pointing
The aim of this fMRI study of prosodic, syntactic,
manual and ocular pointing
is to test the hypothesis that these different pointing
modalities could activate neighbouring cerebral areas.
Lœvenbruck et al. (2009)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Material
� 3 Tesla MR imager (Bruker)
� Overt speech: Oral productions recorded using an fMRI compatible microphone
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Noise reduction: - Microphone outside scanner, at one end of a wave guide
- Other end of guide connected to a mask placed over subject’s mouth
Subject training
Inside the tunnel: the subject wears a mask and can see the stimuli on the rear-view mirror
Cardboard tunnel and paper video-projection screen
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
3T fMRI session
The subject wears the mask and can see the stimuli on the rear-view mirror.
The elbow lies on an arm rest during manual pointing, finger movements are monitored using a digital camera (outside the room, behind window).
Eye-movement are monitored using an ASL 504 eye-tracker during ocular pointing.
� 15 right-handed speakers of French
(9 women and 6 men), aged 18-55 yrs
� 4 sessions: - prosodic focus (vocal pointing)
- syntactic extraction (vocal pointing)
- index finger pointing (digital pointing)
- eye pointing (ocular pointing)
� event-related fMRI paradigm
� Each session : 5 conditions (pointing vs. control, preparation vs. execution, null event) lasting 4.5s each
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Underlying question: - Does Lise hold the book?
- No, it’s JULES who holds the book.
= pointing with the eye, the index or the voice
-Yes.
= no pointing
Prosodic modality Syntactic modality
Control “Lise tient le livre.” “Lise tient le livre.”condition “Lise holds the book.” “Lise holds the book.”
Pointing “JULES tient le livre.” “C’est Jules qui tient le livre.”Condition “JULES holds the book.” “It’s Jules who holds the book.”
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Prosodic modality Syntactic modality
Control “Lise tient le livre.” “Lise tient le livre.”condition “Lise holds the book.” “Lise holds the book.”
Pointing “JULES tient le livre.” “C’est Jules qui tient le livre.”Condition “JULES holds the book.” “It’s Jules who holds the book.”
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Digital modalitymodality Ocular modality
Control
downward finger movement downward look
Pointing finger pointing eye pointingtowards Jules towards Jules
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Control condition utterance
Thanks to the mask, the sentence pronounced by the subject (Thanks to the mask, the sentence pronounced by the subject (““LiseLise tienttient le le livrelivre””) is clearly audible, despite the ambient noise (115dB). ) is clearly audible, despite the ambient noise (115dB). F0 tracking is even possible.F0 tracking is even possible.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Prosodic pointing utterance
An F0 rise on “JULES” and a post-focal deaccentuation are observed
here, typical of a deictic intonation.
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Syntactic pointing utterance
Digital pointing
- Superior parietal lobule (BA 7)bilaterally, with left predominance
- Inferior parietal lobule (BA 40) bilaterally, with left predominance
- Left frontal cortex (BA 4, 6)
LLRR
Ocular pointing
- Left postcentral gyrus (BA 3)
- Bilateral frontal lobe (BA 4, 6)
- Bilateral occipital areas (BA 17, 18)
LLRR
Prosodic pointing
- Superior parietal lobule (BA 7) bilaterally, with left dominance
- Left postcentral gyrus (BA 2, 3)
- Bilateral perisylvian areas (BA 47, 13, 42)
LLRR
Yellow : digital
Cyan : ocular
Red : prosodic
In the superior parietal lobule:
Green: digital + ocular
Purple: ocular + prosodic
Syntactic pointing
- No left parietal activation (right BA 40 only slightly activated)
- Right perisylvian areas (BA 44, 47, 41)
- Bilateral premotor cortex (BA 6)
x
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Discussion
- Activation of left superior parietal lobule (BA7 and neighbouring BA
3) in manual, ocular and prosodic pointing.
- Ocular pointing more anterior than prosodic pointing, itself more
anterior than digital pointing.
- Speakers may use multisensory (auditory, somatosensory)
representations to produce prosodic pointing, just as they do in
manual and ocular pointing. These representations may be formed
via the activation of associative parietal areas.
- Non-significant parietal activation during syntactic pointing
(grammaticalisation = frozen construction = no multisensory
representation necessary = deparietalisation)
DiscussionMultimodal pointing
-acoustic-articulatory -cerebralcorrelates
Vocal pointing:
Manual and ocularpointing
Introduction
Conclusion
These results suggest that the left superior parietal lobule could be a
potential substrate for pointing with the finger, the eye, and the
larynx.
They are in line with our conjecture that linguistic on-line pointing
(prosodic focus) is grounded in the same cerebral network as
gestural (manual and ocular) pointings.
They shed light on the complementary relationship between syntax
and prosody, with syntactic focus conceived as more ‘frozen’ or
‘stored’ and prosodic focus as more ‘built on-line’.
Many thanks to all our participants and to:
MonicaBaciu
Christian Abry
Marion Dohen
LaurentLamalle
ChristophSegebarth
Jean-Luc Schwartz
Coriandre Vilain
CédricPichat
Marie-AgnèsCathiard
FrancescaCarota
Pascal Perrier
Christophe Savariaux
A handful of thanks to Jean-François Dars, a point photographer