Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics...
Transcript of Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics...
![Page 1: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/1.jpg)
Mireille BessonEquipe Langage, Musique et Motricité
Institut de Neurosciences Cognitives de la Méditerranée
CNRS-Marseille
BrainTalk2 - 4 june, 2008 Lund, Sweden
Thanks to the Organizers!
Electrophysiological studies of prosody
![Page 2: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/2.jpg)
Outline
o Modality and semantics (Astésano et al, 2004)
� On-line processing of modality
� Modality and Semantics : interactive processing
o Prosodic focus and pragmatics (Magne et et al, 2005)
� on-line processing of focal accents
� Processing of focal accents interacts with discourse context
o Meter and semantics (Magne et al, 2007)
� on-line processing of syllabic lengthening
� Meter and Semantics: interactive processing
![Page 3: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/3.jpg)
![Page 4: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/4.jpg)
Emotional function:Emotional function: express joy, sadness, anger express joy, sadness, anger ……
(Schirmer et al, 2002; Kotz et al, 2003, (Schirmer et al, 2002; Kotz et al, 2003, …… ))
![Page 5: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/5.jpg)
Experiment 1
Men Women
F5 F6
P5 P6
F5 F6
P5 P6
-5 µV
800 ms
N400
Experiment 2
Men Women
F5 F6
P5 P6
F5 F6
P5 P6
-5 µV
800 ms
N400
Schirmer et al, Cognitive Brain Research, 2002
SOA 100 ms SOA 300 ms
![Page 6: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/6.jpg)
Emotional function:Emotional function: express joy, sadness, anger express joy, sadness, anger ……
(Schirmer et al, 2001; Kotz et al, 2003, (Schirmer et al, 2001; Kotz et al, 2003, …… ))
Linguistic function:Linguistic function: modality , focus, segmentation, modality , focus, segmentation, …… through intonation, accents,through intonation, accents,meter, pauses meter, pauses (Alter at al, 2003; 2005; 2007; Astesano et al, 2003; B(Alter at al, 2003; 2005; 2007; Astesano et al, 2003; Bööcker et cker et al, 1999; Eckal, 1999; Eckstein & Friederici, 2005; Friedrich et al, 2004; Magne et al, 2stein & Friederici, 2005; Friedrich et al, 2004; Magne et al, 2005; 005; Meyer et al, Meyer et al, 2000; Steinhauer et al, 1999; 2000; Steinhauer et al, 1999; ……))
![Page 7: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/7.jpg)
From Steinhauer, Alter & Friederici, 1999. Nature Neurosciences
![Page 8: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/8.jpg)
Emotional function:Emotional function: express joy, sadness, anger express joy, sadness, anger ……
(Schirmer et al, 2001; Kotz et al, 2003, (Schirmer et al, 2001; Kotz et al, 2003, …… ))
Linguistic function:Linguistic function: modality , focus, segmentation, modality , focus, segmentation, …… through intonation, accents,through intonation, accents,meter, pauses meter, pauses (Alter at al, 2003; 2005; 2007; Astesano et al, 2003; B(Alter at al, 2003; 2005; 2007; Astesano et al, 2003; Bööcker et cker et al, 1999; Eckal, 1999; Eckstein & Friederici, 2005; Friedrich et al, 2004; Magne et al, 2stein & Friederici, 2005; Friedrich et al, 2004; Magne et al, 2005; 005; Meyer et al, Meyer et al, 2000; Steinhauer et al, 1999; 2000; Steinhauer et al, 1999; ……))
Acoustic parameters:Acoustic parameters: Pitch/FrequencyPitch/FrequencyLoudness/IntLoudness/IntensityensityDuration/TimDuration/TimeeTimbreTimbre
![Page 9: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/9.jpg)
Modality and semantic processing
Astésano, Besson & Alter (Cognitive Brain Research, 2004)
ERP marker of an incongruous modality ?
Time course of modality and semantic processing
Independant vs interactive processes ?
![Page 10: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/10.jpg)
Materials
2,5
3
3,5
4
4,5
5
5,5
6
6,5
7
S1 S2 S3 S4 S5 S6 Syllables
Yes/No
Questions
Statements
ERB Values
F0 contours of the Prosodically Congruous sentences
2,5
3
3,5
4
4,5
5
5,5
6
6,5
7
S1 S2 S3 S4 S5 S6 Syllables
*Questions
*Statements
ERB Values
F0 contours of the Prosodically Incongruous sentences
Le chauffeur conduisait?
The chauffeur was driving?
La lumière clignotait.
The light was flashing.
*Le piéton traversait.
*The pedestrian was crossing.
*Le cycliste pédalait?
*The cyclist was pedaling?
![Page 11: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/11.jpg)
Procedure
16 French native speakers
2 judgment tasks
�Attention to Semantic: congruous/incongruous
�Attention Prosody: congruous/incongruous
Base Line Auditory Stimuli Yes/No Answer
‘ Le garcon coloriait’ (The boy was drawing) XXX
200 ms 2000 ms 2000 ms
� 28 scalp electrodes (International 10/20 system)
![Page 12: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/12.jpg)
![Page 13: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/13.jpg)
![Page 14: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/14.jpg)
![Page 15: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/15.jpg)
![Page 16: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/16.jpg)
![Page 17: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/17.jpg)
Modality and Semantics
P800 : ERP marker of modality processing
Semantic processing precedes modality processing(but differences in task difficulty)
Interaction between semantic and prosodic processing
![Page 18: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/18.jpg)
Prosodic focus
Magne, Lacheret, Morel, Alter & Besson
(Jal of Cognitive Neuroscience, 2005)
Discourse : pragmatic structure organized around 2 main concepts:
topic : known information (le thème = what we are speaking about)
focus : new information /relevant (le rhème = what we are saying about
it)
Focus is marked :
prosodically : focal accent F0, duration and intensity
![Page 19: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/19.jpg)
Objectives
Are focal accents processed in real time?
Time course of focal accent processing as afunction of pragmatic constraints
![Page 20: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/20.jpg)
Stimuli
Questions :
Q1 Tu as eu une bonne ou une mauvaise note à ton contrôle d’histoire ?
Q2 Où as tu eu une mauvaise note, à ton contrôle d’histoire ou d’anglais ?
Answers :
Focus on sentence middle word (M)R1 En fait, j’ai eu une mauvaise note à mon contrôle d’histoire
Focus on sentence terminal word (T).R2 En fait, j’ai eu une mauvaise note à mon contrôle d’histoire
![Page 21: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/21.jpg)
4 experimental conditions
Focus on sentence middle or terminal words (M vs T)
Focus at good or bad positions (+ vs -)
M+(T0) Focus good on middle word (none on terminal word)
M-(T0) Focus bad on middle word (none on terminal word)
T+(M0) Focus good on terminal word (none on middle word)
T-(M0) Focus bad on terminal word (none on middle word)
![Page 22: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/22.jpg)
Task: is the intonation of the answer correct in the context?
EEG acquisition
| Question | ISI | Phrase Réponse | XXXX |1000 2000 1000 msec
|-------------------|-------|---------*--------*----------------------|----------------|--------| essai suivantM T
PROCEDURE
Participants: 16 native French speakers; right handed
![Page 23: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/23.jpg)
Behavioural data
Conditions M+ M- T+ T-
% Erreurs (Sd) 5 (5) 9 (6) 5 (6) 6 (5)
![Page 24: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/24.jpg)
Sorry for the delay.
With focal accent
Without focal accent
Congruous
Incongruous
Cz
Pz
Cz
Pz
Middle word
- 2 µV + 2 µV
300-1000 ms
ms
5 µV
![Page 25: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/25.jpg)
3 0 0 3
300-450 ms 450-600 ms 600-750 ms 750-900 ms
3
900-1050 ms
a. Focal accent (M- - M+)
b. No focal accent (M0T- - M0T+)
30 3 3 3
300-450 ms 450-600 ms 600-750 ms 750-900 ms 900-1050 ms
0
150-300 ms
0
150-300 ms
Pragmatic congruence
(difference waves)
+ 3.5 µV- 3.5 µV 0
![Page 26: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/26.jpg)
Congruent
Incongru
Cz
Pz
Cz
Pz
Terminal word
With focal accent
Without focal accent
- 2 µV + 2 µV
300-1000 ms
5 µV
ms
![Page 27: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/27.jpg)
300-450 ms 450-600 ms 600-750 ms 750-900 ms 900-1050 ms
a. Focal accent (T- - T+)
b. No focal accent (M-T0 – M+T0)
300-450 ms 450-600 ms 600-750 ms 750-900 ms 900-1050 ms150-300 ms
150-300 ms
Pragmatic congruence
(difference waves)
+ 3.5 µV- 3.5 µV 0
40 0 0 0 3
00 0 3 3 3
![Page 28: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/28.jpg)
ConclusionsConclusions
o Focal accents are processed in real time
� Psychobiological relevance of prosodic focus
o Interaction between prosodic focus and pragmatic context
� Pragmatic relevance of focal accent determines the observed effects (more than its presence - absence)
o Observed effects depend upon the position of focal accents in the sentence
� Middle word : positivity Surprise
� Terminal word : negativity Integration
![Page 29: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/29.jpg)
Meter and semantic processing
Magne, Astésano, Ystad, Kronland-Martinet & Besson(Cerebral Cortex, 2007)
ERP marker of incongruous syllabic lengthening ?
Independent vs interactive processes ?
Influence of attention
![Page 30: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/30.jpg)
Meter
Semantic
Congruous(M+)
Incongruous(M-)
Congruous (S+)M+S+
Le concours a regroupémille candidats
M-S+Le concours a regroupé mille
candidats
Incongruous (S-)M+S-
Le concours a regroupémille bigoudis
M-S-Le concours a regroupé mille
bigoudis
32 sentences / experimental condition (= 128 sentences)
Experimental conditions
![Page 31: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/31.jpg)
Task Meter : Is the final word well-pronounced ?
Semantic : Is the final word meaningful in the context?
Participants: 14 native French speakers, right-handed
Protocole
Acquisition EEG + % errors + RTs Time course
| Phrase | Mot final | XXXX |200 2000 2000 ms
|------------------*------|-----------------|----------------|- … Essai suivantMarker
![Page 32: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/32.jpg)
Mètre Sémantique
S+M+ S-M+ S+M- S-M- S+M+ S-M+ S+M- S-M-
%Err 13 (16) 19 (16) 15 (14) 17 (16) 11 (6) 15 (17) 24 (18) 15 (11)
TRs 932 (152) 1055 (179) 966 (102) 997 (135) 975 (139) 1127 (184) 1011 (164) 1138 (146)
Behavioural data
![Page 33: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/33.jpg)
Cz
Pz
SemanticMeter
S+M+
S-M+
N400 N400
1000 ms
-5 µV
300-600 ms 600-1000 ms 600-1000 ms300-600 ms
![Page 34: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/34.jpg)
Cz
Pz
S+M+
S+M-
P700
N400
1000 ms
-5 µV
300-600 ms 600-1000 ms300-600 ms 600-1000 ms
SemanticMeter
![Page 35: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/35.jpg)
Cz
Pz
S+M+
S-M-
N400N400
P700 1000 ms
-5 µV
300-600 ms 600-1000 ms 300-600 ms 600-1000 ms
SemanticMeter
![Page 36: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/36.jpg)
ConclusionsConclusions
o P700 to metric incongruity when attention is focused on meter
� participants are sensitive to the metric structure of words
o N400 to metric incongruity when attention is focused on semantics
� Interaction between metric and semantic processing
o N400 to semantic incongruity independently of attention.
� N400 also reflects automatic semantic processing
![Page 37: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/37.jpg)
![Page 38: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/38.jpg)
![Page 39: Mireille Besson Equipe Langage, Musique et Motricité ......Outline o Modality and semantics (Astésano et al,2004) On-line processing of modality Modality and Semantics : interactive](https://reader034.fdocuments.us/reader034/viewer/2022052011/6026fe499e6ce1631c3ca942/html5/thumbnails/39.jpg)
o Psychobiological validity of several aspects of prosodic
processing :
Modality
Focal accent
Meter
o Interactions between levels of processing :
Prosodic
Semantic Pragmatic
o Linguistic specificity ?
MusiqueLangage