Mémoire, Apprentissage et Langage: Aspects Cognitifs

Peter Ford DOMINEY

Institut des Sciences Cognitives, CNRS UMR 5015, UCB Lyon 1

Plan• Langage: Constructions grammaticales• Structure des séquences sensorimotrices• Réseau Neuronal• « Equivalence » apprentissage langage et

séquences abstraites• Bases neurophysiologiques• Ré-éducation• Apprentissage multi-langue• Apprentissage robotique• Fin

Constructions Grammaticales et mots de fonction:

Same meaning different syntactic structure Different transformation from sentence to meaning Signaled by function words (the, was, to, by) Function words trigger transformation/mapping rules to

associate open class nouns to their thematic roles (agent, object, recipient).

The dog gave the elephant to the monkey.

Gave (Agent, Object, Recipient)

The elephant was given to the monkey by the dog.

Structure Abstraite et Constructions Grammaticales

• Constructions Grammaticales– Mapping <Forme – sens>

• Langage– Ensemble structuré de

constructions grammaticales

• Correspondance entre– Constructions grammaticales

&– Structures abstraites

• Hypothèse d’Équivalence

Forme: NP1 VT NP2 à NP3

Sens: ActT(Agent, Obj, Recip)

Forme: Jean donne la balle à Marie

Sens: Donne(Jean, Balle, Marie)

Construction Di-transitif

J D B M – D J B M

Structure Abstraite « Di-Transitif »

Dominey Evolution of Communication 02,Dominey et al., Brain and Language 03

J D B M

D J B M

Structure des Séquences Sensorimotrices

1. Sérielle (ordre)– ABCBACB– CABACBA

2. Temporelle (rythme)– A BCAB AC A– AB C A BAC A

3. Abstraite (règle)– ABC-BAC– DEF-EDF– ….– 123 – 213

Selection

WMABCR

eco

g

State

Output

Input

Cortex

RecurrentConnections

StriatumReward-related dopamine synaptic plasticity

A B C

BAC

Prefrontal Working Memory

1. Dominey et al. JCogNeuro 95, Dominey BioCybernetics 95Dominey & Boussaoud CogBrRes 97

2. Dominey JEP 98, Dominey CogBrainRes 983. Dominey Brain&Lang 97, Dominey et al. JCogNeuro 98, Dominey & Ramus Lang & Cog Proc. 2000, Dominey et al. 1997,

Lelekov et al. 2000

Réseau Neuronal

123

213

Model learns to perform thematic role assignment

Training: Present sentence

one word at a time Indicate: agent,

object and recipient Learn

transformational relations so State can Modulate appropriate STM to Output

Testing & Correction: Present new

sentences Model emits the

correct roles Error-Learning

100% Correct Caplan

StateSelection

Output

Input

The elephant was given to the monkey by the dog.

Closed ClassStream

(Function):

the, was, to, by

Open ClassStream (Content):

Canonically ordered nouns:agent(dog), object(elephant),

recipient(monkey)

STMelephant

monkeydogR

eco

g

Dominey, et al, Brain and Language, 2003.

Theoretical Implications

• Functional relation between linguistic and non-linguistic abstract sequence processing

• Common neurophysiological basis for rule-based structural transformation processing

ABCx

Closed ClassStream

(Function):x

Open ClassStream (Content):A,B,C

StateSelection

Output

STM/Rec

Input

Symbols in transformed order:BAC

The elephant was transferredto the monkey by the dog.

Closed ClassStream

(Function):the, was, to, by

Open ClassStream (Content):elephant, dog, monkey

Canonically ordered nouns:agent(dog), object(elephant), recipient(monkey)

StateSelection

Output

STM/Rec

Input

Corrélats Neuro-Linguistiques:chez des patients aphasiques agrammatiques

1. Perturbations Corrélées entre Séquences Abstraites - Langage

2. Transfert d’apprentissage entre Séquences Abstraites - Langage

Syntax

Abs

trac

t Seq

0

4

8

12

16

20

14 18 22 26 30 34 38 42

r 2 = 0.86p = 0.003 R

L

Cor

rect

of 5

1

2

3

4

5

Active Passive Relative

PrePost

P = 0.02

1. Perturbations Corrélées

Lelekov, Dominey et al. 00; Dominey et al. Brain&Language 03, Hoen et al. NeuroReport 03

2. Transfert d’apprentissage

Corrélats Neuro-Linguistiques:chez des patients aphasiques agrammatiques

Tracé de Moyennes

2°ordre intéract.

F(4,20)=3.19; p<.0352

Beh

avio

r

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Pre-test Mid-test Post-test

WORD ORDER PRODUCTION SYNTACTIC COMPREHENSIONSEQUENTIAL ORDER TRANSFORM

Language Abstract Sequences

ERP• Common Cortical Network• Langage/Abstract Seq • LAN, P600, N400• ABCXBAC vs ABCYABC

(Michel Hoen - Thesis 2003)

Neurophysiology - ERP:Neurophysiology - ERP: Shared cortical networks for abstract sequence and language

Compare Sentence and Sequence Processing for closed class elementsSentence: • The faucet was soldered by the plumberSequence: • ABC X BAC

Corrélats Neurophysiologiquesen IRM fonctionnelle

• Réseau cortical commun pour le traitement des transformations structurelles

• Réseau cortical distinct pour l’intégration de la semantique dans ces transformations

Newman 2003

IRM

f

Langage Langage & Seq

Hoen, Dominey et al. Cortex 2005 –special issue on “Integrative models of Broca’s area and the ventral premotor cortex”

Modèle Actualisé: Traitement des Constructions Grammaticales

IRM

f

donnéMarieballeJean

ACTIONAGENT

OBJECTRECIPIENT

“__ a été __ à __ par __”

balledonnéJeanMarie

balledonnéJeanMarie

“La balle a été donnéà Jean par Marie”

OpenClassArrayPredictedReferents Array

SceneEventArray

ConstructionIndex

ConstructionInventory

SentToScene WordToReferent

Langage & Séquence(Pars opercularis) Ba 44Transformations Structurelles

Langage spécifique(Pars triangularis) Ba 45Intégrations Sémantiques

B A C

“__ __ __ X __ __ __

A B C

ABC X BACConstructionIndex

ConstructionInventory

Dominey & Hoen Cortex 2005 Special Issue on Integrative Models of Broca’s Area

Langage Séquences

Grammar and Grammatical Constructions Learned

• 1. S NP + VP• 2. NP Det + N• 3. NP NP+Rel+ VP• 4. VP Va + NP• 5. VP Va + NP + PP• 6. VP Aux + Vp + PP • 7. VP Aux + Vp + PP + PP• 8. PP Prep + NP• 9. Det the, a• 10. N cylinder, block, moon• 11.Va touched, pushed, took, gave• 12.Vp touched,pushed, taken, given• 13. Aux was• 14. Prep to, by, from• 15. Rel that

• The block pushed the triangle.• The block gave the triangle to the moon.• The triangle was pushed by the block.• The moon was given to the triangle by

the block.• The block that pushed the triangle

touched the moon.• The block pushed the triangle that

touched the moon.• The block pushed the triangle that was

touched by the moon.• The block that was pushed by the

triangle touched the moon.• 35+ syntactic structures

Cross-Linguistic Validation: Japanese

• Free word order

• Grammatical roles of NPs indicated by post-positions

• Will the “cue competition” model adapt to this configuration?

• -ga, -ni, -yotte, -to and –wo markers as closed class elements

Cross-Linguistic Validation: Japanese

Block-ga circle-wo oshita triangle-ni-yotte tatakareta.Circle-wo oshita triangle-ni-yotte block-ga tatakareta.The block was hit by the triangle that pushed the circle.Pushed(triangle, circle), Hit(triangle, block) Circle-wo tataita block-ga triangle-ni-yotte osareta.Triangle-ni-yotte circle-wo tataita block-ga osareta.The block that hit the circle was pushed by the triangle.Hit(block, circle), Pushed(triangle, block) Block-ga circle-wo oshita triangle-wo tataita.Circle-wo oshita triangle-wo block-ga tataita.The block hit the triangle that pushed the circle.Pushed(triangle, circle), Hit(block, triangle)

N = 26 constructions

Using These Learned Grammatical Constructions for Event

Description and Interrogation

Humannarrator

Vision, "Meaning"Extraction

Spoken Language Interface

(CSLU RAD)

Grammatical Construction Model: Sentence to Meaning

The moon gave the cylinder to the block.The block was gave the cylinder by the moon.The cylinder was gave to the block by the moon.

Event(Agent, Object, Recipient)

E(A,O,R) Sentence

CCD Camera

Gave(moon, cylinder, block)

Grammatical Constructions

for Robot Command• Grammatical constructions

– Link between command meaning (predicate-argument structure) and sentence structure

– Sentence - Meaning mappings• Event description

– Action(Agent, object, recipient)

– Sentence

• Commanding– Sentence

– Move(Object, location)

Plan• Langage: Constructions grammaticales• Structure des séquences sensorimotrices• Réseau Neuronal• « Equivalence » apprentissage langage et

séquences abstraites• Bases neurophysiologiques• Ré-éducation• Apprentissage multi-langue• Apprentissage robotique• Fin