CognitionPsychology 215

Emory UniversityLawrence W. Barsalou

Topic 9: LanguageLecture 9b: Language Comprehension

Language comprehension 2

In-class video and questions

Language comprehension 3

The traditional view:Detached amodal representations underlie language comprehension

• comprehension in classical views of cognition

Language comprehension 4

The emerging view:Simulations of being there underlie language comprehension

• the purpose of language comprehension is not to simply store text information

• situated functions of comprehension

• the applicability of simulation in comprehension for situated action

Language comprehension 5

Types of evidence for the emerging view

• perceptual simulation• increasing evidence suggests that perceptual simulations represent meaning

• mental models• these simulations are mental models of the corresponding situations

• perspective• these mental models are viewed from a perspective

• situated action• language comprehension appears to serve situated action• it’s not just text memory

Language comprehension 6

Orientation in comprehensionStanfield and Zwaan (2001)

• hypothesis• if readers simulate the meaning of a text to understand it,

then orientation effects should occur when matching sentences to pictures

• method• a sentence was presented• a picture was presented of something that was or was not mentioned in the sentence• subjects indicated as quickly as possible whether what the picture represented was present

• key manipulation• whether or not the orientation of the pictured object matched the orientation of the object

implied in the sentence

Zwaan

Examples of sentences Examples of pictures

Mary pounded the nail into the wall(implies a horizontal nail)

Mary pounded the nail into the floor(implies a vertical nail)

False responseTrue response

Language comprehension 7

ResultsStanfield and Zwaan (2001)

• conclusions

800

825

850

875

900

Consistent Inconsistent

Picture orientation

Verif

icat

ion

Tim

e (m

sec)

Language comprehension 8

Shape consistency in comprehensionZwaan, Stanfield, & Yaxley (2002)

• hypothesis• if readers simulate the meaning of a text to understand it,

then other perceptual factors besides orientation should be important in processing it• for example, the shapes of objects

• method• a sentence was presented• a picture was presented and subjects had to name it as quickly as possible

• key manipulation• whether or not the pictured object had a shape that matched the shape of the object implied

in the sentence

Examples of sentences Examples of pictures

The bird sat quietly in the tree(implies a bird with its wings folded)

The bird flew quickly across the sky(implies a bird with its wings flapping)

True response False response

Language comprehension 9

ResultsZwaan, Stanfield, & Yaxley (2002)

• conclusions

600

610

620

630

640

650

Consistent Inconsistent

Picture shape

Nam

aing

Tim

e (m

sec)

Language comprehension 10

• mental model• a simulation of the spatial situation that a discourse is about, and the events taking place there• e.g., when reading a text about an office, the reader should simulate the office space and events

• prediction• if people simulate mental models, spatial distance should affect comprehension

• i.e., because mental models have spatial distance, just like their referent situations in the world• if the reader must switch attention from one location in the model to another,

the farther the distance between the two locations, the longer comprehension should take• analogous to scanning effects in perception and imagery (Lecture 5b)

• Bower and Morrow review a long series of studies done in the 1980s

Bower

Mental models in language comprehensionBower and Morrow (1990)

Language comprehension 11

MethodBower and Morrow (1990)

• subjects memorized a map of a physical setting (e.g., an office building)• rooms, relations between rooms, and objects in rooms• other experiment describe the layout in a text, not with a map

• subjects then read a story about the setting• a character who wandered through the building doing various things

• critical tests• at random points in the text, received a pair of objects

projector - clock (true)projector - lamp (false)

• subjects had to indicate as quickly as possiblewhether the two objects were from the same room

• manipulation• on true trials, whether the two objects were from:

• the goal room (the room just entered)• the source room (the room just left)• another room (in the same building)• another building

• prediction• the distance from the character’s current location

to the two objects should determine response time

Language comprehension 12

ResultsBower and Morrow (1990)

• distance effect

• intermediate path effect

Language comprehension 13

Another test of mental modelsGlenberg, Meyer, and Lindem (1987)

• issue• same spatial assumptions about mental models as Bower and Morrow (1990)

• if subjects construct mental models to represent the meaning of texts, spatial distance should be important

• Glenberg et al. test this assumption in a different way• manipulate whether or not an object is associated with a character’s body as s/he move about

• when the object is associated, spatial distance is small, and the object is verified quickly• when the object is not associated, spatial distance is large, and the object is verified slowly

• method• subjects read a text• at random points, they received a probe word

and indicated whether it occurred in the text or not• the text manipulated whether the target object was

associated or unassociated with the character’s body• the distance between the probe and target was

manipulated by 0, 1, or 2 filler sentences

• prediction• if subjects construct a mental model of the text,

the associated texts should produce faster verification,especially as the delay grows, and the character movesfurther in distance from the initial location

Glenberg

Language comprehension 14

ResultsGlenberg, Meyer, and Lindem (1987)

• distance effect

1 vs. 2 scenariosis whether thecharacter stayswithin 1 situationor moves to asecond situation

Language comprehension 15

• subjective perspective• when subjects simulate mental models while comprehending texts,

they may adopt the subject perspective of the focal character• if so, texts consistent with the perspective of the main character should be easier to comprehend

than texts inconsistent with it

• method• subjects read texts, and the time to read sentences was measured• consistent versus inconsistent perspective was manipulated between sentence clauses:

Bill was sitting in the living room reading the paper (consistent)when John came into the living room.

vs.Bill was sitting in the living room reading the paper (inconsistent)when John went into the living room.

Fred was just sitting down by the fire (consistent)when his faithful dog brought him his slippers.

vs.Fred was just sitting down by the fire (inconsistent)when his faithful dog took him his slippers.

camevs.

went

broughtvs.

took

Black

Subjective perspective in language comprehensionBlack, Turner, and Bower (1979)

Language comprehension 16

ResultsBlack, Turner, and Bower (1979)

• perspective effect

• bias for recalling a consistent perspective

0

10

20

30

40

Consistent Inconsistent

Subjective perspective

P(sw

itch

pers

pect

ive)

280

300

320

340

360

380

Consistent Inconsistent

Subjective perspective

Rea

ding

tim

e pe

r syl

labl

e (m

sec)

Language comprehension 17

More evidence for perspectives in comprehensionSpivey, Tyler, Richardson, and Young (2001)

• “being there” during comprehension• Spivey et al. argue that readers put themselves in the situation that a text describes to

understand it

• method• subjects wear an eye tracking helmet that records the positions of their eye focus• however they believe that it has been turned off while they take a break• during the break, they’re asked to listen a story, and to give an opinion on it• the story varies in the perspective that the main character takes on a focal object

• key manipulation• the position of the focal object relative to the main character

• UP: the main character is looking up from the street at an event on top of a skyscraper• DOWN: the main character is looking down into a deep canyon at an event

• prediction• if readers experience “being there” as part of comprehension,

their eye movements should drift up during the skyscraper story, and down during the canyon story

Spivey

Language comprehension 18

ResultsSpivey, Tyler, Richardson, and Young (2001)

0.0

0.1

0.1

0.2

0.2

Relevant Irrelevant

Eye direction

P(ey

e m

ovem

ent)

• conclusions

Language comprehension 19

Glenberg KaschakMotor simulations during comprehensionGlenberg & Kaschak (2002)

• motor simulations represent language about action

“Pick up the book.”

• method• subjects read a sentence and judged whether it was grammatical or ungrammatical• indicated grammatical vs. non-grammatical with a forward vs. backward lever push,

• for half the subjects, a forward push (away) indicated grammatical • for the other half, a backward push (toward) indicated grammatical

• manipulation and prediction• whether the direction of a comprehended action was consistent or inconsistent with

the lever push•consistent: “Open the drawer” with a backward (toward) lever push• inconsistent: “Open the drawer” with a forward (away) lever push

• if subjects simulate the action to comprehend it, a consistency effect should occur

Language comprehension 20

ResultsGlenberg & Kaschak (2002)

1700

1750

1800

1850

Away Toward

Sentence meaning

Judg

men

t tim

e (m

sec)

Away TowardResponse direction

• conclusions

Language comprehension 21

Conclusions

• perceptual simulation• increasing evidence suggests that perceptual simulations represent meaning

• mental models• these simulations are mental models of the corresponding situations

• perspective• these mental models are viewed from a perspective

• situated action• language comprehension appears to serve situated action• it’s not just text memory

Language comprehension 22

Bibliography

Barsalou, L.W. (1999). Language comprehension: Archival memory or preparation for situated action? Discourse Processes, 28, 61-80.

Black, J.B., Turner, T.J., & Bower, G.H. (1979). Point of view in narrative comprehension, memory, and production. Journal of Verbal Learning and Verbal Behavior, 18, 187-198.

Bower, G.H., & Morrow, D.G. (1990). Mental models in narrative comprehension. Science, 247, 44-48.Glenberg, A.M., Meyer, M., & Lindem, K. (1987). Mental models contribute to foregrounding during text

comprehension. Journal of Memory and Language, 26, 69-83.Glenberg, A.M., & Kaschak, M.P. (2002). Grounding language in action. Psychonomic Bulletin & Review,

9, 558-569. Marslen-Wilson, W.D., & Tyler, L.K. (1980). The temporal structure of spoken language understanding.

Cognition, 8, 1-71.Potter, M.C., Kroll, J.F., Yachzel, B., Carpenter, E., & Sherman, J. (1986). Pictures in sentences:

Understanding without words. Journal of Experimental Psychology: General, 115, 281-294.Potter, M.C., & Lombardi, L. (1990). Regeneration in the short-term recall of sentences. Journal of Memory

and Language, 29, 633-654.Spivey, M., Tyler, M., Richardson, D., & Young, E. (2000). Eye movements during comprehension of

spoken scene descriptions. Proceedings of the 22nd Annual Conference of the Cognitive Science Society, (pp.487-492). Mahwah, NJ: Erlbaum.

Stanfield, R.A., & Zwaan, R.A. (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science, 12, 153-156.

Zwaan, R.A., Stanfield, R.A., & Yaxley, R.H. (2002). Language comprehenders mentally represent the shapes of objects, Psychological Science, 13, 168-171.