Post on 23-Feb-2016
description
Brain related semantics
CSCTR – Session 8Dana Retová
Rejects the standard view that amodal symbols represent knowledge in semantic memory
Cognition shares the same mechanisms with perception, action and introspection
Simulation◦ A core form of computation in the brain◦ Reenactment of perceptual, motor and introspective
states acquired during experience
◦ As experience occurs, the brain captures the states across modalities and integrates them with a multimodal representation stored in memory
Grounded cognition
Modal representation and imagery representing knowledge◦ Epicurus, Kant, Reid
Behaviorists◦ Imagery not sufficiently scientific
Cognitivists◦ Amodal representation (feature lists, semantic
networks, frames) Elegant and powerful formalisms for representing
knowledge Could be implemented in AI
Mental imagery vs. amodal symbols
No evidence supports the presence of amodal symbols in cognition
Grounding problem◦ Traditional theories fail to explain how cognition
interfaces with perception and action Problem where the brain stores amodal
symbols◦ How is it consistent with neural principles of
computation?
Problem with traditional AI
Simulation Situated action Bodily states
◦ Modal representations are central to knowledge
Grounded cognition theories
Cognitive Linguistics Theories◦ Lakoff & Johnson (1980, 1999)
Abstract concepts are grounded metaphorically in embodied and situated knowledge
Theories of situated action◦ Gibson (1979)
Role of environment in shaping cognitive mechanisms Coupling of perception and action during goal achievement Social interaction
◦ Research in robotics◦ Dynamic systems as preferred architecture
Fixed representations do not exist
Theories of Grounded Cognition
Memory theories◦ Glenberg (1997)
Memory is not just passive storage of information Perception of relevant objects triggers affordances for
action stored in memory Reasoning about future actions relies on remembering
affordances while suppressing perception of the environment
Social simulation theories◦ How we represent the mental states of other people
We use simulations of our own minds To feel someone else’s pain we simulate our own pain Mirror neurons Empathy, imitation, social coordination
Cognitive Simulation Theories
Perceptual Symbol Systems◦ Synthetic approach
Implements standard symbolic functions Type-token binding, inference, productivity, recursion, propositions
◦ A single multimodal representation system in the brain supports diverse forms of simulation across different cognitive processes High-level perception Working memory long-term memory conceptual knowledge
Convergence zone architecture (Damasio 1989, Simmons & Barsalou 2003) Single representation system controlled by multiple simulation
mechanisms
Cognitive Simulation Theories
ensemble of neurons within which many feedforward/feedback loops make contact.
It 1) receives forward projections from cortical regions located in the connectional level immediately below
2) Sends reciprocal backward projections to the originating cortices
3) Sends forward projections to cortical regions in the next connectional level; and
4) Receives projections from heterarchically placed cortices and from subcortical nuclei in thalamus, basal forebrain, and brainstem.
Convergence zone
Perceptual Inference Perception-action coordination Perception of space Memory
◦ Implicit memory◦ Explicit memory◦ Working memory
Conceptual processing
Empirical Evidence for Grounded Theories
Vision and motion◦ Goldstone (1995)
Association between shape and color◦ Hansen et al. (2006)
Object’s natural color distort achromatic perception of the object toward the opponent color
◦ Motion (Freyd 1987, Shiffrar & Freyd 1990,1993) Subjects simulate the visual trajectory beyond its actual
trajectory Also during apparent motion, simulation of possible
action shapes the perception of motion◦ Speech (Warren 1970) : http://www.youtube.com/watch?
v=UlJs24j3i8E Lexical knowledge produces simulation in speech
perception – missing phoneme simulation
Perceptual inference
Simulations of potential actions◦ Viewing an object grasped with a precision or power grip (grape
vs. hammer) produces a simulation of the appropriate action (Tucker & Ellis 1998) This is affected by
object’s orientation (Symes et al. 2007) Size (Glover et al. 2004)
◦ Simulations of both grasping and functional actions (Bub et al 2007)
◦ Also name triggers simulation (Tucker & Ellis 2004)◦ Hearing a word activates the articulatory action associated with
producing it (Pulvermuller 2006)◦ Perceived effort affects visual perception (Proffitt 2006)
Being tired from a run makes a hill look steeper Carrying a heavy pack makes a path look longer
Perception-action coordination
Motor simulations◦ Motor system constructs a feed-forward
simulation of the action to guide and correct it (Grush 2004, Wolpert et al. 1999)
◦ Generating visual inferences about the anticipated actions of perceived agents (Wilson & Knoblich 2005)
Perception-action coordination
The perception of space is shaped by the body and it’s relation to the environment◦ Locating objects has various difficulty along
different axes Vertical
easiest Front-back Left-right
Most difficult – bodily cues are lacking◦ Perception of near space extends with arm length
(Longo & Laurenco 2007)
Perception of space
Results form simulation of perceptual memories
Repetition priming is strongest when the modalities of the memory and stimulus match (e.g. auditory) (Kirsner et al., 1989)
Repetition priming is strongest when perceptual details of the memory and stimulus match (e.g. orientation, size,…) (Jacoby&Hayman, 1987)
Imagining produces repetition priming similar to actual perception (Roediger&Blaxton, 1987)
Implicit memory
Multimodal simulations of previous episodes◦ Important for constructing future events
The retrieval of a word stimulates the modal operations performed at encoding (Wheeler et al. 2000)◦ Visual areas become active during retrieval
following visual study while auditory areas become active following auditory study
Greater activation in modal areas when remembering something that really occurred than false memories (Slotnick & Schacter 2004)
Explicit memory
Absent stimulus is stored in working memory (Levy & GoldmanRakic 2000)◦ To maintain working memory, neurons in the frontal lobes
maintain a simulation of the absent stimulus in the modal system that processed it originally. Some frontal regions maintain working memories of
objects, other spatial locations, motion, textures, etc. They are highly selective for the specific features
Visual imagery in working memory simulates visual processing (Finke 1989, Kosslyn 1980,…)◦ Analogously, motor imagery, auditory imagery, etc.
Mental rotation of visual objects -> motor simulations of making them turn (Richter et al. 2000)
Working memory
Behavioral evidence◦ When asked whether an property belongs to an objects
subjects simulate properties to verify them (Solomon & Barsalou 2004)
Lesion evidence◦ Lesions in one modality – losing categories that rely on it
for processing (Damasio 1994, …) E.g. damage to visual areas – losing of ability to
categorize animals (visual processing is dominant) Damage to motor areas – categorization of tools
Neuroimaging evidence (Martin 2001, 2007)◦ When processing conceptual knowledge, brain areas
representing properties are active
Conceptual processing
Perceptual simulation Motor simulation Affective simulation
Language comprehension
Situation models◦ Evidence of modal representations in language
comprehension Spatial representation (Bower & Morrow 1990) People confused pictures with text (Intraub &
Hoffman 1992) Replacing words with pictures did not disrupt
sentence processing (Potter 1986)
Language comprehension
Subjects read a sentence and then processed a picture that either matched or mismatched something implied by the sentence◦ “The ranger saw the eagle in the sky”◦ Picture of an eagle – wings outstretched or folded
Visual irrelevant information interferes with spatial inferences (Fincher-Keifer 2001)
Perceptual simulation
Verbs for head, arm and leg actions produce head, arm and leg simulation in the respective areas of the motor system (Pulvermuller 2005)
When action to make a response is consistent with text meaning, the response is quicker (Glenberg & Kaschak 2003)
Subjects simulate corresponding motion through space (Richardson et al. 2003)
Positive/negative valence (Meier & Robinson)
High/low power (Schubert 2005)
Motor simulation
Subjects’ faces configured according to sentences with emotional content (Havas 2007)
When facial emotion matched the content comprehension was better
Gesture◦ Producing gestures helps speakers retrieve words
whose meaning are related to the gestures (Krauss 1998)
◦ Also help listeners comprehend what speaker says◦ Children can gesture before speaking
Affective simulation
Physical reasoning◦ Gear, pulleys◦ Driven by spatial simulation◦ Sketchy, not holistic and detailed
Abstract reasoning◦ Content effects◦ Reasoning about time using space domain -
metaphors
Thought
Embodiment effects◦ Activating elderly stereotype causes people to
walk slowly and to perform lexical decision slowly (Dijksterhuis & Bargh 2001)
◦ Engaging the smiling musculature produces positive affect (Strack et al. 1988)
Social mirroring◦ Individual differences in the ability to simulate
other people’s mental states correlate with rated empathy (Jackson et al. 2005)
Development◦ Mirroring, object permanence…
Social Cognition
Questions?