How We Cognize Space Zenon Pylyshyn Rutgers Center fir Cognitive Science Rutgers University, New...

How We Cognize SpaceHow We Cognize SpaceZenon PylyshynRutgers Center fir Cognitive ScienceRutgers University,New Brunswick, NJ

How we cognize spaceHow we cognize spaceIs There anIs There an IImagemage S Space in the Head?pace in the Head?

The most common approach to the question of how we represent a spatial layout is that we represent it in the form of a mental image. The format of mental images is supposed to be particularly suited for representing spatial information.

The mind’s great illusion: That you see a world inside your head when you imagine the “Picture Theory” of mental imagery

The four-part plan of this lecture:The four-part plan of this lecture:1. First I will talk a little about “the imagery debate” and

introduce recent neuroscience evidence.2. I will then focus on what I consider the core of the

debate: How mental images represent space.3. Then I will talk about the special case of images that

are projected onto the perceived world. 4. To show the generality of projected images that I will

need to spend a few minutes to introduce the idea of visual indexes or FINSTs, as a type of a deictic or demonstrative reference.

5. Finally I will combine the idea of indexes with the evidence on mental imagery and suggest how generalized indexes may allow spatial properties of the currently-perceived world to translate into apparent spatial properties of images.

I.I. The imagery debate: The imagery debate: A capsule overview A capsule overview

The main question is whether thoughts experienced as “mental images” or as “seeing with the mind’s eye” are different from other thoughts, and if so how.

The dominant view is the “picture theory” of mental images, which assumes that images stored in a spatial medium and are examined by the visual system the same way that the original scene would be.

Does imagery use the visual Does imagery use the visual system and if so, what does that system and if so, what does that tell us about the nature of images?tell us about the nature of images? There is some evidence that the visual system is

“active” during imagery This has led to the view that the visual system

must be examining some not-yet-interpreted image, just as it was thought to do in visual perception.

But the last step is unwarranted because even if the visual system was involved, it would only mean that both vision and imagery use some of the same processes and the same kind of representations, but neither need be pictorial.

Failure of the picture-theory in visionFailure of the picture-theory in visionIn vision the picture theory was meant to explain why our perception is panoramic and stable while the visual inputs are highly local, partial and constantly changing

But the picture theory of vision has been thoroughly discredited: There is no rich panoramic display in vision (e.g., see change blindness, superposition studies, …)

The picture theory of visionThe picture theory of visionis a non-starter, even for catsis a non-starter, even for cats

(Cartoon by Kliban)

A more plausible theory of visionA more plausible theory of vision(even for cats)(even for cats)

IIII. The newest round of . The newest round of the imagery debatethe imagery debate

In recent years the picture theory has been revived, due largely to two neuroscience findings:1. The visual cortex (V1) is activated during imagery2. The visual cortex is retinotopically organized (i.e.,

it appears to map the retina in a topographically continuous or homeomorphic manner).

From this, people have concluded that mental imagery uses a literal spatial display, located in V1.

The goal of neuroscience research on The goal of neuroscience research on mental imagery is to find a display of the mental imagery is to find a display of the imagined pattern in visual corteximagined pattern in visual cortex

Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218(4575), 902-904.

We already know that there is a topographical projection of retinal activity in visual cortex

The tool of choice has been the use of brain scans (esp fMRI, PET)

What do recent neuroscience results What do recent neuroscience results tell us about mental imagery? tell us about mental imagery?

1. Even if there is a 2D mapping of retinal activity in visual cortex (V1), this should not be identified with the mental image.

2. Patterns in V1 do not function the same way as mental images for several reasons.

3. Even if dynamic 3D patterns were found in V1 it would not explain most mental imagery research findings.

None of the brain-scan (fMRI, PET) results None of the brain-scan (fMRI, PET) results supports the picture theory of mental supports the picture theory of mental images for reasons that I will discuss nextimages for reasons that I will discuss next

The topographical structure of the visual The topographical structure of the visual cortex could not support mental imagescortex could not support mental images

1. Even of there is a 2D mapping of retinal activity in V1, this cannot be identified with the mental image which is panoramic, 3-dimensional, dynamic and has many other properties that could not be mapped onto V1, so we would need a different theory for them.

Why activity in visual cortex could Why activity in visual cortex could not correspond to a mental imagenot correspond to a mental image

2. Patterns in V1 are different from mental images: a) Patterns in V1 are foveal and retinocentric while

mental images are panoramic and allocentricb) There is no spontaneous 3D interpretation of

patterns in mental images <parallelogram example>

c) There is no amodal completion of patterns in mental images <Kanizsa example>

d) Order of access of information in a mental image is not free <name letters of a familiar word backwords>

e) Emmert’s law does not hold for images <unlike afterimages>

f) There is no visual (re)interpretation of images <Slezak example>



mental images are panoramic allocentricb) There is no spontaneous 3D interpretation of



d) Order of access of information in a mental image is not free <letter reading example>

e) Emmert’s law does not hold for imagesf) There is no visual (re)interpretation of images

<Slezak example>

Imagine two parallelograms (as Imagine two parallelograms (as below) one above the anotherbelow) one above the another

Close your eyes and then imagine Close your eyes and then imagine these two parallelogramsthese two parallelogramsConnect the corresponding top and bottom vertices

What do you see? Keep looking to see if anything changes

Did you see this? Did it flip?Did you see this? Did it flip?

Amodal completion in imagery?Amodal completion in imagery?

Slezak figuresSlezak figures

Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

Rotated Slezak figuresRotated Slezak figures

No subject was able to recognize the mentally rotated figure

Subjects remembered the figures well enough so if they drew it they could recognize the rotated figure

Even if patterns in visual cortex were Even if patterns in visual cortex were isomorphic to those in the mental isomorphic to those in the mental image, it still would not explain most image, it still would not explain most results of mental imagery research!results of mental imagery research!

3. The reason that patterns of activation in striate cortex would not explain most of the results of mental imagery research is that the results are largely cognitively penetrable and therefore require the appeal to knowledge, goals, utilities, etc and inferences over them. In other words they require a cognitive explanation.

Task Demands and the Task Demands and the tacit tacit knowledgeknowledge explanation explanation

The task of “imagining X” is the task of pretending that you are seeing X and simulating as much of that event as seems relevant to the task using your tacit knowledge about how the event might unfold. The task also requires certain other skills (e.g., estimating time-to-collision, generating time intervals, etc) but it does not require that you use a spatial display.

Examples…

There are many examples showing There are many examples showing that the result that was attributed to that the result that was attributed to the mental image format is actually the mental image format is actually due to tacit knowledgedue to tacit knowledge

Color mixing example to illustrate the difference between the two sources of observations <slide>

Imagine dropping weights from different heights Mental Image size (It has been shown that it

takes longer to report small details from a small image than from a large on. What does this mean? What would you think if the result showed the opposite?)

Mental scanning <example slide>

Color mixing exampleColor mixing example

Studies of mental scanningStudies of mental scanningA window on the mind?A window on the mind?

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Relative distance on image

La

ten

cy

(s

ec

s)

scan image

imagine lights

show direction

(Pylyshyn & Bannon. See Pylyshyn, 1981)

Mental representation of space:Mental representation of space:The core of the imagery debateThe core of the imagery debate

It seems to be almost impossible to deny that thinking using mental images exploits spatial properties of images in some important sense. In what sense? (Do images “preserve metrical spatial properties”?)

It is always possible to encode spatial relations in any form of representation that has a numeral system, so why assume that the representation of space is itself spatial?

1) Phenomenology: we see things as “laid out in space”!

2) Psychophysical evidence from projected images (illusions, S-R compatibility…)

3) Clinical evidence (visual/imaginal neglect)

Use of real visible space in Use of real visible space in “projected” mental imagery“projected” mental imagery

“Projected images” serve to directing attention and to associate thoughts with selected visible objects. Examples: Robust version of mental scanning (scanning with eyes open) Visual illusions involving projected images <Bernbaum & Chung,

1981>

Projected memory images act like displays <Podgorny & Shepherd>

S-R Compatibility with images (Tlauka & McKenna, 1998)

Visuomotor (prism) adaptation from mental images <Finke, 1980>

Shepard & Podgorny experimentShepard & Podgorny experiment

Both when the displays are seen and when the F is imagined, RT to say whether the dot was on the F was fastest when the dot was at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.

S-R Compatibility Effect with displayS-R Compatibility Effect with display

S-R Compatibility Effect with ImagesS-R Compatibility Effect with Images

Might all spatial images Might all spatial images work like work like projected imagesprojected images??

There are three key ideas behind the proposal that spatial mental images are the projection of the spatial layout of imagined objects onto a perceived scene

1. Recognition that the spatial properties exhibited in experiments with projected images depend only on the location of a few items and not on other visual properties

2. The idea of a limited-capacity amodal indexing mechanism or deictic reference: FINSTs and Anchors.

3. The idea of a primitive amodal spatial sense that allows us to perceive and recall the location of objects in an allocentric frame of reference, independent of the objects’ perceptual properties or of sense modality, and automatically updated by our movements

We don’t need a spatial display in our We don’t need a spatial display in our head if we have the right kind of deictic head if we have the right kind of deictic contact with real (perceived) spacecontact with real (perceived) space

None of the experiments that are alleged to show the existence of a spatial display (in visual cortex) need to appeal to anything more than a small number of imagined locations. (e.g., Shepherd & Podgorny, Finke, Tlauka,…)

If we can index a small number of (occupied) locations in real space (using FINSTs) we can use them to allocate attention or to program motor commands.

If these indexed objects are also bound to objects of thought this will result in our thoughts (i.e. images) having persisting spatial relations.

Aside on Visual Index (FINST) TheoryAside on Visual Index (FINST) Theory

FINSTs are direct, unmediated, nonconceptual connections between objects in the world and mental symbolsFINSTs serve as visual demonstratives (like “this” or “that”).

Such direct references are essential for solving the correspondence problem in vision – especially in the case of visual representations built up incrementally over different glances or “noticings”.

Some instances where we need Indexes:Visual stability, recognizing n-place relations, subitizing, and

multiple-object tracking

Several objects must be picked out Several objects must be picked out at once in relational judgmentsat once in relational judgments

When we judge that certain objects are collinear, we must have picked out the relevant individual objects first.

Several objects must be picked out Several objects must be picked out at once in relational judgmentsat once in relational judgments The same is true for other relational judgments like

inside or on-the-same-contour… etc. We must pick out the relevant individual objects first.

A concrete demonstration of A concrete demonstration of what visual indexes can dowhat visual indexes can do

Multiple Object Tracking studies (MOT) Basic finding: People can track up to 5 individual

objects that do not have a unique description We have shown that it is unlikely that the tracking

is done by updating locations but rather that individuating and keeping track of certain kinds of individuals is a primitive visual operation

Tracking is primitive and likely both preconceptual and preattemtive

The mechanism for tracking is the same as the mechanism that is used for picking out elements when images are “projected” onto a scene.

How do we do it? What properties of individual objects do we use?

But you can also imagine in the But you can also imagine in the dark or with your eyes closed!dark or with your eyes closed!

Does imagery work differently in the dark or with eyes closed?

Must indexes be visual?

The Sense of Space The Sense of Space This phrase is meant to denote an extremely well-

developed human capacity to recall and orient to locations in space; a space that is independent of modality and is anchored to real allocentric space.

There is a major difference between a sense of space and a visual image. The sense of space is not a subjective experience but a skill that is largely unconscious. There has long been a suspicion that what has been studied under the name “mental imagery” is really spatial ability (e.g., unconscious images?).

The sense of space does not need an internal spatial medium; it can derive spatial properties by binding mental particulars to real perceived space.

Perceptual Indexes (I.e., FINSTs and Anchors) are mechanisms that allow representations to inherit some of the spatial properties of the perceived world.

Some illustrations of the sense of spaceSome illustrations of the sense of spaceMany phenomena that have been cited in support of the Many phenomena that have been cited in support of the picture theory of mental imagery only implicate a picture theory of mental imagery only implicate a spatialspatial

sense, not the visual perception of a mental displaysense, not the visual perception of a mental display

Sense of space is not specific to (or parasitic on) visionBlind people exhibit all the observed phenomena of mental

imagery Responses to images exhibit S-R compatibility and the

Simon effect – i.e., reactions made towards a stimulus are faster than ones made away from it.The space that is relevant to the Simon effect is amodal (you get

cross-modal Simon effects) Hemispatial Neglect is a deficit in orienting attention to

real locations – that’s why it may be mirrored in imagery Mental Images can induce visuomotor adaptation

But only location, not visual pattern, plays a role (R. Finke) Observations such as the mental scanning effect, when

they are not due to task demands, can be explained in terms of scanning through perceived space

ConclusionConclusion Many of the “mental imagery” findings in the literature are the

result of subjects using their tacit knowledge to simulate what it would be like to see the situation described.

The neuroscience evidence does not show that there is a 2D display in visual cortex on which we “draw” images when we imagine. The activity in visual cortex is of the wrong kind to underwrite mental imagery.

More interesting are the studies in which people project images onto perceived scenes because these studies do show the involvement of spatial properties. But these experiments never need to assume that a picture-like pattern is projected. All they need to assume is that a few objects in the visual scene are indexed and associated with objects of thought. The rest of the spatial properties come from perception.

Although the clear cases are when images are projected onto a visual scene, the same is likely true of other modalities that contribute to our sense of space.

Representing spaceRepresenting space

The spatial character of mental images (and other “spatial” representations) comes from binding objects of thought to real objects in 3D space. The space in mental imagery comes from real concurrently-perceived spatial relations, which give us our exquisite sense of space.

ReferencesReferences

Bernbaum, K., & Chung, C. S. (1981). Müller-Lyer illusion induced by imagination. Journal of Mental Imagery, 5(1), 125-128.

Kanizsa, G., & Gerbino, W. (1982). Amodal completion: Seeing or thinking? In B. Beck (Ed.), Organization and Representation in Perception (pp. 167-190). Hillsdale, NJ: Erlbaum.

Pylyshyn, Z. W. (1973). What the Mind's Eye Tells the Mind's Brain: A Critique of Mental Imagery. Psychological Bulletin, 80, 1-24.

Pylyshyn, Z. W. (1981). The imagery debate: Analogue media versus tacit knowledge. Psychological Review, 88, 16-45.

Pylyshyn, Z. W. (1989). The role of location indexes in spatial perception: A sketch of the FINST spatial-index model. Cognition, 32, 65-97.

Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 197-207.

Pylyshyn, Z. W. (2001). Visual indexes, preconceptual objects, and situated vision. Cognition, 80(1/2), 127-158.

Pylyshyn, Z. W. (2002). Mental Imagery: In search of a theory. Behavioral and Brain Sciences, 25(2), 157-237.

Pylyshyn, Z. W. (2003). Return of the Mental Image: Are there really pictures in the brain? Trends in Cognitive Sciences, 7(3), 113-118.

Pylyshyn, Z. W. (in press). Seeing and visualizing: It's not what you think. Cambridge, MA: MIT Press/Bradford Books.

Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spatial Vision, 3(3), 1-19.

Shepard, R. N., & Metzler, J. (1971). Mental rotation of three dimensional objects. Science, 171, 701-703.

Shepard, R. N., & Podgorny, P. (1978). Cognitive processes that resemble perceptual processes. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (Vol. 5). Hillsdale, NJ: Erlbaum.

Slezak, P. (1995). The `philosophical' case against visual imagery. In P. Slezak, T. Caelli & R. Clark (Eds.), Perspective on Cognitive Science: Theories, Experiments and Foundations (pp. 237-271). Stamford, CT: Ablex.

Tlauka, M., & McKenna, F. P. (1998). Mental imagery yields stimulus-response compatibility. Acta Psychologica, 67-79.

How We Cognize Space Zenon Pylyshyn Rutgers Center fir Cognitive Science Rutgers University, New...

Documents

Transcript of How We Cognize Space Zenon Pylyshyn Rutgers Center fir Cognitive Science Rutgers University, New...