Premotor cortex, action control, and...

Premotor cortex, action control, and language

Arthur M. Glenberg

University of Wisconsin-Madison

Ago, ergo Cogito

Action control is amazinghitting a tennis ball playing piano walking down steps

Ago, ergo Cogito

hitting a tennis ball, playing piano, walking down steps

coordination among muscles - hierarchical controlover serial order

continuous recalibration

exquisitely precise timingexquisitely precise timing

slow sensory feedback and irrelevant feedback

Evolution and the brain have done a wonderful job solving a set ofvery tricky problems.

Might those solutions be used to solve other tricky problems?

Today’s talk

Ago, ergo CogitoA solution to action control problems: controller/predictor models of action control, Wolpert’s HMOSAIC t ti i tHMOSAIC as a starting point.

Relating HMOSAIC to the brain: Canonical and Mirror Neurons

A marvelous confluence: The HMOSAIC model of action control,with minimal changes, can be a model of language learning,

d ti d h iproduction, and comprehension.

Data from several research projects testing the claim that languagp j g g gcomprehension is intimately related to action control.

Wolpert’s MOSAIC

feedbackAgo, ergo Cogito

lift 150 g

socialenvironmentpredictorlift 500 g

lift 350 g

GainControl

Predictions &probabilities

environment

body/perceptual

predictorsensory/envpredictor

perceptualstates

h i l

controllercontroller +

physicalenvironment

efference copypypredictor

faster than feedbackselects relevant feedback (attention)

feedback & errorlearning predictor selects relevant feedback (attention)

helps learning controllerenhances perception

glearning controllerlearning responsibilitie

drink

feedback: move eyesfeedback: extendHMOSAIC

Ago, ergo Cogitosocialenvironment

sensory/envpredictor

drink

Gain<post<post.

body/perceptualcontroller

predictor


GainControleyes,…>

i

extend,…>

perceptualstates

controller <prioreyes,…><priorextend,…>

physicalenvironment

+efference copyeyesgraspextend

efference copy

t d grasp lifteyes extendbasic motor repertoire; motor cortex:

feedback

HMOSAIC and language

Ago, ergo Cogito

i ldi t

drink

feedback

socialenvironment

predictor

Predictions &

GainControl

body/perceptual

states

controller


physicalenvironment

+grasp

efference copy

canonical neuronsmirror neurons

canonical neurons

Ago, ergo Cogitomirror neurons

purposeaction recognitionintent/theory of mindenhance perceptionenhance perceptioncommunication?

Human datafMRI, TMS,

behaviorBroca’s & handBroca s & handBuccino, Tettamanti,Aziz-Zadeh, more

Fadiga et al., speech MNFrom, Rizzolatti & Arbib, 1998

feedback

HMOASIC and languageBroca’s area: speech and hand; Fadiga et al. speech MN

Ago, ergo Cogito

i ldi t

drink

feedback

/ “drink”

socialenvironment

predictor

Predictions &

GainControl

speech

body/perceptual

states

controller


hBroca’s area

physical

speech

bi directional assoc

control over serialorder in speech

environment

+grasp /dr/bi-directional assocof speech and actio

gesture

efference copy

canonical neuronsmirror neurons

grounding of meaning

feedbackLearning nouns

socialenvironment

predictor

bottle / “bottle”Ago, ergo Cogito

environment

body/

speechPredictions &probabilities

GainControl

body/perceptual

states

controller

speech

p

Assume child knows how to interact with a

physicalenvironment

how to interact with a bottle: move eyes, grasp

Joint attention: eyes+ Speech mirror neuron

graspeyes

Canonical neurons

efference copyphonology & meaning ->

a construction

feedback

Learning verbs

Ago, ergo Cogito

i ldi t

drink / “drink”

socialenvironment

predictor

speechPredictions &

GainControl

body/perceptual

states

controller

h


physical

speech

Assume childknows how to drinkenvironment

+Auditory mirror&

grasp /dr/

knows how to drink

efference copy

&Action mirror

Dad says, “Give Dada the cup.”

feedback

Ago, ergo Cogito

di t

give cup / “give”social

predictor

speechPredictions &

GainControl

environment

controller

h

probabilities body/perceptual

statesAssume child knows

“cup” ->

speech

physicalenvironment

how to give a cup

peyesgrasp

Action

releaseextendgrasp +Action

efference copy

give5/“give”i 4/“ i ” 3 “Give Lavinnia the candy ”

5 “Give Dada the spoon”4 “Give Dada the candy”

In other words: Behavior consistent with having learned an abstract, double-object construction emerges from integrating (by responsibilities)give4/“give”

give3/“give”give2/“give”

Ago, ergo Cogito2 “Give Momma the bottle.”3 Give Lavinnia the candy.

1 “Give Dada the cup.”feedback

emerges from integrating (by responsibilities) actions learned for specific, concrete situations. Syntax (and compositionality) emerges from

ti t lgive2/ give

give1/“give”Learning speech/action paramepredictor

feedback

socialenvironment

action control.

Attend with eyes to speaker

Attend with eyes to object,controller

speech Predictions &probabilities

GainControl body/

perceptualstates

Attend with eyes to speaker

Attend with eyes to object, set kinematics, graspspeech

states

physicalenvironment

+

Now “Give Dada the bottle”

y pspeaker

object

graspExtend arm to speaker

+

controller

predictorspeech

bottle / “bottle”

Now, Give Dada the bottlespeaker

act

controllerspeech

graspeyes

The Data!

Ago, ergo CogitoLanguage acquisitionAngrave & Glenberg (2007)

Action systems affect language comprehensionGlenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (200Glenberg & Cattaneo (in preparation)

Language comprehension affects action systemsBorghi, Glenberg, & Scorolli (in preparation)

A & Gl b (2007)

Does the child know about bottles before learning “bottle?”

Ago, ergo Cogito

Angrave & Glenberg (2007)Analysis of MacArthur-Bates CDI for gesture & speech

• A01 armshow Look • C14 pushtoycarPush• A01. armshow Look • A02. reachout Look • A03. points See

A09 requestarm Get

• C14. pushtoycarPush • C15. throwball Throw • E01. sweep Sweep

E02 keyindoor Put• A09. requestarm Get • A12. Shrugsallgone Finish • B02. pattycake Clap

B05 sing Sing

• E02. keyindoor Put • E03. Poundhammer Hit • E06. pretendread Read

E09 pla m sinstr Pla• B05. sing Sing • B06. dance Dance • C01. Eatspoon Eat

C02 d i k D i k

• E09. playmusinstr Play • E10. pretenddrive Drive • E13. writepen Write

E15 t l P t• C02. drinkcup Drink • C05. wipeownface Clean

• E15. putonglasses Put

28

A12Verb onset month = .61 x Action onset month + 17 monthsp < .001

26E01 E02

E13E15

pwrite

mon

th

24

tesnhtnom

A01A02

A09B02

B05

B06C05

C14C15 E03

E10clap throw

sweep

rb o

nset

m

22

breVO

A03

E06E09

drinkread

Ve

20

C01

C02 read

4 6 8 10 12 14 16Action Onset monthAction onset month

feedback

Why the lag between comprehension and production?

Ago, ergo Cogito

i ldi t

drink / “drink”

socialenvironment

predictor

speechPredictions &

GainControl

body/perceptual

states

controller

h


physical

speech

environment

+grasp /dr/

efference copy

Controller and predictor for Art’s knees

Ago, ergo CogitoFatigue muscles systemsfeedback

Bean task!

socialenvironmentpredictor

step

G i

/“step”

environment

body/perceptual

statescontroller


GainControl

speech

states

physicalenvironment

+step /st/

speech

efference copy

+step /st/

efference copy

2500

Con Sent PNCon Sent Tow (Art gave you the pen)You see the coins in the fountain

John see the coins in his hand.

2250

mse

c)

Con Sent Far

Abs Sent Away (You told Liz the story)

Abs Sent Tow (Liz told you the story)

Con Sent Away (You gave Art the pen)You see the coins in the fountain.

2000

men

t Tim

e (m

y ( y)

1750

Judg

m

Con Sent YouCon Sent Near

You see the coins in your hand.You see the coins in your hand.

1500

12501250 Tow Away Tow Away Tow Away Tow Away Tow Away

beans lexical decision beans beans beansExp 1 Exp 2 Exp 3 Exp 3 Exp 3

Conclusions

In designing primate action systems, evolution figured Ago, ergo Cogito

g g p y , gout solutions to difficult problems of timing, learning, serial order, and hierarchical control.

Many of those same problems occur in language, and it appearsthat evolution adapted mechanisms of action control, in Broca’s at lto language.

Data supporting these claims come frompp glanguage acquisitionaffects of action systems on language comprehensionaffects of language comprehension on use of action systemaffects of language comprehension on use of action system

Language can be used as a tool because linguistic meaning i d t i d b d t d t b dil ti i th ldis determined by and connected to bodily action in the world.

Bologna collaborators: Anna Borghi, Claudia Scorolli, Annalisa SettiLeipzig collaborators: Shirley-Ann Rüschemeyer, Wolfgang Prinz, Simone Bosbach

Many Thanks to:

The Wisconsin Laboratory for Embodied Cognition:

Ago, ergo Cogito

Parma collaborators: Mark Sato, Luigi Cattaneo, Daniele Palumbo, Lucia Riggio, Givoanni Buccino, Vittorio Gallese, Giacomo Rizzolatti

The Wisconsin Laboratory for Embodied Cognition:Doctoral Students: Post-doc:

Megan Brown Lawrence AngraveDavid HavasDavid Havas

Erica Kesin RichmondLisa Lindeman

Undergraduate Students:gJenna Dittmar Laura KleinTiana Gutierrez Stephen MlinazLauren Hazen Josh MoonMaia Jacobs Emily MouilsoBeth Jaworski Michal Rischall Scott Johnson

Funding: National Science Foundation, University of Wisconsin, Max Plank Institute for Human Cognitive and Brain Sciences, & University of Parma, Department of Neuroscience

Iacoboni (2005)

Ago, ergo Cogito

= predictor= controller controller

lenberg, Sato, Cattaneo, Riggio, Palumbo, & Buccino (2007)

Ago, ergo Cogito

Trans-cranial magnetic stimulation (TMS)

TMS DesignTransfer vs No-Transfer vs NonsenseConcrete vs AbstractPulse at Verb or End

record MEP from OPrecord MEP from OPNB: NO ARM MOVEMENT FOR RESPONSE

Ago, ergo Cogito

0.4 TransferNo Trans

0.2

0

-0.2

Concrete Abstract

-0.4Verb End Verb End

Pulse LocationPulse timePulse time

Language comprehension affects motor systems

Hamilton, Wolpert, & Frith (2004) “Your own action

Ago, ergo Cogito

influences how you perceive another person’s action”lift 15 kg

lift 150 gfeedback from vision

lift 150 g

Predictions &

socialenvironmentpredictor

/lift 500 g

lift 350 g

GainPredictions &probabilities

body/perceptual

statescontroller

sensory/envpredictor

+

GainControl

states

physicalenvironment

controller +

efference copy

Borghi, Glenberg, & Scorolli (in preparation)

Videos from Bosbach, Cole, Prinz, & Knoblich

Ago, ergo CogitoLarge box weights:3 6 12 18 kg

, ,(2005)

3, 6, 12, 18 kgSmall box weights.05, .3, .6, .9 kg

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.Heavy lift: Move the bowling ball from the cart to the tablefrom the cart to the table.Light lift : Move the wine glass from the cart to the table.Descriptive: The wine glass is on thetable.Is the object on the table breakable?

Ago, ergo Cogito

1on

0.75

ion

Hea Video

Cor

rela

tio 0.5

Cor

rela

ti

Light Video

Heavy Video

C

0.25

lift lift

0Descriptive Heavy Light

lift liftSentence Type

Rüschemeyer, Glenberg, Kaschak, & Friederici

Ago, ergo CogitoTable 1Examples of the sentences with English translations

Towards Away StationaryDas Auto fährt auf Dich zu Das Auto Fährt von Dir weg Das Auto sieht großDas Auto fährt auf Dich zu.Das Auto Fährt von Dir weg. Das Auto sieht großThe car drives towards you. The car drives away from you. The car looks bigDer Pfeil fliegt auf Dich zu. Der Pfeil fliegt von Dir weg. Der Pfeil sieht klien aus.Th fli t d Th fli f Th l kThe arrow flies towards you. The arrow flies away from you. The arrow looks Der Ball rollt auf Dich zu. Der Ball rollt von Dir weg. Der Ball sieht groß aus.The ball rolls towards you. The ball rolls away from you. The ball looks big.

Ago, ergo Cogito

“Give momma the bottle.”“Give momma the bottle.”Dad says,

Ago, ergo Cogito

di t

give / “give”feedback

socialpredictor

speechPredictions &

GainControl

environment

“momma” ->eyes

controller

h

probabilities body/perceptual

states

“bottle” ->

reachspeech

physicalenvironment

eyesgraspreleaseextendgrasp

eyes to mommaeyes to bottlegain control gain control+

gain controlact!

efference copyreleaseextendgrasp

Language comprehension & simulation“The girl gives the horse an apple.”

give3/“give”give2/“give”

Ago, ergo Cogito

speech activates motor systemgive1 /“give”

feedback

p y

gain control set

“give”-> speech MN & controller

predictor

speech

give1 / give

P di ti &Gain

Control

socialenvironment

give -> speech MN & controllerpredictor generates

expectations of a recipiencontroller

h


Controlbody/

perceptualstates

speech

physicalenvironment+

lh

“the horse”-> speech MN & conpredictor generates

sensory feedback of horse

Efference copy -> A mental modelwith perspective

applehorse sensory feedback of horse

with perspectiveEmergence of working memory

eyes grasp

“Transfer X the Y.”

Ago, ergo Cogito

Learning kinematicspredictorhand/“hand”

feedback

socialenvironment g

Hand X the Y. (hand priors)Th X th Y (th i )controller

predictor


GainControl

environment

body/perceptual

Throw X the Y. (throw priors)controller

speech

perceptualstates

physicalenvironment

Result: Easily learned action schemas for verbs of concrete

gainlocate X

locate Y

environment+

schemas for verbs of concrete transfer.

locate Y

kinematics

gain Syntactic/semantic bootstrappingusing the motor system!

“Transfer X the Y.”

Ago, ergo Cogito

Learning kinematicspredictortell/“tell”

feedback

socialenvironment g

Tell X the Y. (tell priors)Radio X the Y (radio priors)controller

predictor


GainControl

environment

body/perceptual Radio X the Y.(radio priors)

Delegate X the Y. (delegate prioetc.

controller

speechstates

physicalenvironment

Results: Easily learned action schema for

gainlocate X

locate Y

+

Easily learned action schema for verbs of abstract transfer.

locate Y

kinematics

gain

1400Ago, ergo Cogito

1300

1400No Transfer

Away Toward

ACE interaction

1200

1300 ACE interaction& main effect of C t1200 Concreteness

1100

Concrete Abstract

Motor system forunderstanding

or1000

Response Location

Yes-near Yes-farYes-near Yes-farMotor imagery &responding

Response Location

Ago, ergo Cogito

The Plan:

Replicate Glenberg & Kaschak using Italian

Use TMS to determine if action effect is during understanding or responding.

Collaborators: Marc Sato Luigi Cattaneo LuciaCollaborators: Marc Sato, Luigi Cattaneo, Lucia Riggio, Daniele Palumbo, and Giovanni Buccino

Dipartimento di Neuroscienze, Università di Parma

Triad of Concrete sentences

Toward: Marco/ ti/ da' /le carte. (Marco gives you the papers.)

Ago, ergo CogitoAway: Tu/ dai /le carte/ a Marco. (You give the papers to Marco.)

No Transfer: Tu/ leggi/ le carte/ con Marco. (You read the papers with Marco.)

Triad of Abstract sentences

Toward: Anna/ ti/ delega/ le responsabilità. (Anna delegates the responsibilities to

you )you.)

Away: Tu/ deleghi/ le responsabilità/ ad Anna. (You delegate the responsibilities

to Anna.)

No Transfer: Tu/ discuti/ delle responsabilità/ con Anna. (You discuss the

responsibilities with Anna.)

Nonsense Sentences

Giovanni/ canta/ le carte/ con te. (Giovanni sings the papers with you.)

Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)Daniele/ ti/ pulisce/ la responsabilità. (Daniele cleans the responsibilities to you.)

Note: The slashes indicate the segments used in Experiment 2.

Simulate temporal course of giving (away) vs receiving

t fi t l tAgo, ergo Cogito

at first laterAway: grasp open

Toward: open grasp

Pulse at EndPulse at Verb

AbstractConcreteAbstractConcrete0 5

0.75

p g p

AbstractConcreteAbstractConcrete

0.25

0.5

0

0 5

-0.25

-0.5Toward Away No Trans Toward Away No Trans Toward AwayNo Trans Toward AwayNo Trans

Sentence Direction

Action and the embodiment of concrete language

Ago, ergo CogitoHauk, Johnsrude, & Pulvermüller (2004)

Tettamanti et al (2005)Tettamanti et al. (2005)

Aziz-Zadeh et al. (2006)

What about the embodiment of abstract language?Space:Space:Richardson, Spivey, Barsalou, & McRae (2003) Boroditsky and Ramscar (2002)

Action: Gl b & K h k (2002)Glenberg & Kaschak (2002)

Other Predictions

Ago, ergo Cogito

eye movements and language

development of gesture & languagep g & g g

signed languages

eye movements and languageTanenhaus visual world paradigm

Ago, ergo Cogito

Tanenhaus visual world paradigmAltmann & Kamide

Richardson & colleagues

P dd t l (i )Ago, ergo Cogito

Padden et al. (in press)

In Al-Sayyid Bedouin Sign Language (ABSL), signs for both yy g g g ( ) gconcrete and abstract verbs of transfer involve arm movemeof away from the body.

( )

Other signed languages?

Karen Emmorey (personal communication) “it is true for ASL and probably for many signed languages”

Conclusions

Ago, ergo CogitoThe data strongly imply that concrete language (about

transfer) is grounded in the motor system

The data suggest that abstract language (about transfer)is also grounded in the motor systemis also grounded in the motor system

Models of language built on models of motor controlModels of language built on models of motor controlprovide a unique understanding of languagephonology, syntax, and semantics as arising f hi hi l h i f t t lfrom hierarchical mechanisms of motor control

Why might abstract language understanding involvemotor system? A developmental hypothesis:

Ago, ergo Cogitolearning nounsattention (move eyes); affordances

i l ?

learning verbs of transfer: parameters of actionsource object kinematics (force direction mode)

canonical neurons?

source, object, kinematics (force, direction, mode)mirror neurons?

generalization of parametersgeneralization of parameterssource, objects, kinematics --> transfer schema

syntactic/sematic boot-strapping using the motor system!y pp g g yYou tell Liz the story.

“Give me the cup.”“Give momma the bottle.”“Give Lavinnia the candy ”

Ago, ergo Cogito

Give Lavinnia the candy.

Learning action parameters:predictor

give/“give”

feedback

socialenvironment g p

Gain control (move only eyes)controller

predictor


GainControl

environment

body/perceptual

Attend with eyes to destination{momma, me, Lavinnia}

controller

speech

perceptualstates

physicalenvironment

gainlocate X

locate Y

Attend with eyes, set kinematic{bottle, cup, candy}

environment+

locate Y

kinematics

gainGain control (ready to move)

Result: A double object action schema (verb argument construction) for giveAct Now!

Linda Smith (2005) “Action alters shape categories”

Ago, ergo Cogito

Action

0.9

1

0 5

0.6

0.7

0.8

Sy

0.2

0.3

0.4

0.5As

0

0.1

S1 S2 S3 A1 A2 A3

Similarity of Test Object to Exemplar

Thus, perceived similarity, categorization, and namingare influenced by action.

Glenberg & Kaschak 2002

Action systems affect language comprehension

Ago, ergo Cogito

Glenberg & Kaschak, 2002

Task: Judge if a sentence is sensibleIV 1: Implied Direction: Toward vs. AwayIV 2: Sentence type

ImperativeOpen the drawer. / Close the drawer.

Concrete TransferConcrete TransferCourtney handed you the notebook. / You handed Courtney the

notebook.Andy delivered the pizza to you. / You delivered the Pizza to Andy.

Ab t t T f tAbstract Transfer sentences Liz told you the story. / You told Liz the story.The policeman radioed the message to you. / You radioed the

message to the policeman.g pNonsense

Open the plate./You drank the house to Joe.

IV 3: Actual Response Direction: Yes-is-near vs Yes-is-IV 3: Actual Response Direction: Yes is near vs. Yes isfar

e

1400Imperative

1850

Abstract Transfer

Concrete Transfer

1850A

n R

eadi

ng T

ime

13501800

1750

1800

AwayToward

AwayToward

AwayToward

Tri

mm

ed M

ean

13001700

17501750

1700

Toward

You told Liz the stor

T

1250Yes-is-near Yes-is-far



Response Direction

Is the effect caused by comprehension processes or response pro

Premotor cortex, action control, and...

Documents

Transcript of Premotor cortex, action control, and...