Action Causes Perception Causes Action:From Sensory Substitution to Situated

Robots

Lecture 1+2, Unit 5NUCOG Seminar: Action, Perception, Motivation

Akureyri, Iceland 10.2.-20.2.2006

Marieke RohdeCentre for Computational Neuroscience and Robotics

University of Sussex

Essays

Students:

• Please sign up soon

• If you did tell me make sure I have you on my list

Marking:

• Reviews vs. Original Thinking

• Complexity of the Topic

• Structure/Organisation

• Form: Length, References, …

• No Punishment of Language Difficulties!!!

This module

Today:

1. History and Motivation

2. The Importance of Situatedness: Empirical Evidence

3. A Sensorimotor Account

Thursday:

4. Robotics

5. The Question of Value

6. Conclusion

1.) History and Motivation

The Birth of Cognitive Science

• Dominant psychological Theory in 1950s: Behaviourism

• Cognitive Science, Dartmouth Conference 1956

(Chomsky, Newell, Simon, Minsky, Miller, ...)

• „Cognition as Information Processing“ Metaphor: Ensuring a

scientific framework

Alternative Views

• Cybernetics: We will learn more about

on thursday

• Russian Psychology and Physiology

• Ethology

• ...

Symbolic Cognitivism is a dream of

separation (Perception, planning,

action)

Situated and Embodied Cognitivism is a

dream of integration.

Reciprocally Causal Processes

• General hypothesis: many times,

when you have complex,

durable processes with

circular or reciprocal

causality, you will observe the

formation of some spontaneous

invariant organisation.

• Examples:

– Pheromone trails

– Stigmergy (“historical

processes”)

Reciprocally Causal Processes

Model by Helbing et al. Nature, 388, pp 45 – 50, (1997).

Reciprocally Causal Processes

• Sensorimotor coordination Can be

viewed as a process of reciprocal

causation. But it’s difficult to be

aware of it.

• Fluid behaviour becomes less and

less conscious as it starts to rely

on coherences of SM

coordination.

• Surprise originates only when SM

coordination is broken. (An

unexpected obstacle, etc.)

• Sensorimotor invariants in behaviour

and perception:

– Preferred postures (Draw circle in the

air)

– Perceptual invariants (e.g. size and

colour constancy)

– No flow associated with voluntary eye

movements

2.) Empirical Research

Disease, Sensory Substitution, Perceptual Perturbation

• Functionalist explanations of cognitive behaviour talk – implicitly or

explicitly – about the hazards we encounter and manage as being

predicted

• They assume behavioural programs (written in „genetic code“)

inside the agent/brain.

• Such mechanisms can function only in established situations that

rely on phylogenetic or ontogenetic constancies.

• When established relations break down, things become interesting:

We find changes in the process, that change the relations, and

changes in relations that change the process (Reciprocal causality)

Experiments with Goggles

• Inversion goggles

– Radical disruption, initial

helplessness

– Gradual adaptation and

recovery

– Relies on action and purpose

– Subjective experience

recovers

• We will learn much more about

this in the afternoon.

• (Classroom demonstration)

Recent replication by Sekiyama, Nature, 407, pp 374-377, (2000)

Colour Fringes and Curvature vs. Squinting Goggles

• Prism Goggles:– Adaptation to colour

fringes and line curvature

– But not to stereo-colour effects!

Pictures taken from:

Kohler, I. (1962)

Sensory Substitution

• TVSS:

– Visual tasks mastered after short training.

– Externalisation

• “The use of one human sense to receive

information normally received by another

sense” (Kaczmarek, 1995 with Bach-y-Rita).”

Sensory?

• (Lenay et Al. 2003) Not sensory:

– Retinal/cochlear implants useful but not innovative

– No perception without action:

• Static: no externalisation, just tickling

• Variations in sensations are related to actions: constitution of a new sensory

modality in the adult

• Tool to study plasticity of brain as well as consciousness and intentionality

• Sensory substitution rests on Information processing methaphor, where you

passively receive through arbitrary channels

– Sensorimotor substitution or perceptual substitution

Substitution?

• (Lenay et Al. 2003) Not substitution:

– Abusive term, suggests remedy of a deficit

– A new space of coupling between a human being and

the world (not like vision…) without shared qualia

– Perceptual Supplementation or Perceptual

Augmentation

First Person Experiences

• Visually handicapped persons express disappointment, because they are

unable to give a "content" to the perception (colour, emotional value)

• natural perception != simple capacity to discriminate and categorize.

• Same in operated blind

• Compiegne Hypothesis: Shared meaning has a cultural context and a

personal history.

• Mine and Ezequiel’s Hypothesis: A significance for the subject + more time

may be enough

• Origin and the nature of the value attached to things can be experimentally

investigated.

Space and Externalisation

• 1 sensor, 1 stimulator, on/off

Space and Externalisation

• Blocked elbow and wrist:

direction but no distance

• No variation in direction:

Height and width but no distance

• Not “in” but “in front of the space (bidimensional)

• Construction of a space requires:

– Anticipation

– Reversibility

– Continuity

Space and Externalisation• The proximity of the lightbulb is bound to the speed with which b must be augmented such that a can be reduced

• However, two “contacts” with the lightbulb are not sufficient to give the experience of a distal lightsource.

• The perception of an external object requires a permanent activity that makes the tactile sensation come and disappear constantly.

Classroom Demonstration

• The amateur minimal perceptual supplementation

My own experiences…

– Discovery Zone:

• TVSS shape

recognition

• Eye displacement

• Auditory delay

• Visual inversion

• “Be a robot”

• Cheltenham Festival of Science (June 2005, with S. Angliss, S. McGregor

and B. Bigge)

– Introduce sensory substitution/distortion research to public

– Test bed for me and my colleagues

– “The cyborg experiment”: “Retro man” and “Metal man”

Newspaper Article by Roger Highfield, Daily Telegraph, 25.5.2005

Cyborg Footage

• In the beginning

• After a couple of days

Auditory Delay

• Demo:

– How comes we perceive things as coincident?

(Multimodal) Integration of “The Present”

• Interesting empirical findings:

– Recalibration of Audiovisual Simultaneity (worthwile replicating with

emphasis on motion in its own right)

– No adaptation to auditory/visual delays in reading/drawing tasks

– Cunningham et Al. (2001): old studies did not actually induce lag.

– Eagleman et Al.: Illusory reversals in the timing of actions and sensation

Kittens and Ferrets

• Held (1965)– Kitten Caroussel

• Swindale (2000)– Rewired Ferrets:

• Neuroscientific• Behavioural

What‘s the point?

• The principles of sensorimotor coordination are far from

unshakable

• What is invariant is meaningful behaviour

• Meaning can be created from „unnatural“ sensorimotor

relationships

• To me it is hard to see how a dedicated mechanism of

adaptation, enclosed in a part of the highly variable

sensorimotor loop, could restore meaningful behaviour

3.) A Sensorimotor Account

Noë, Hurley, O‘Regan

• Alternative descriptive Concepts

– Sensorimotor Contingencies (3rd person)

– Cortical Deference/Dominance (1st person)

• Explaining Consciousness (closing the

explanatory gap)

Explanatory Gap

• Levine (1983): Functionalism can describe material causes,

regularities and mechanisms of experience, but it cannot explain the

phenomenal character of experience (ultimate epistemological

obstacle)

• „Qualia are meant to be properties of experiential states or events.

But experiences, we have argued, are not states. Our claim, rather,

is that it is confused to think of the qualitative character of

experience in terms of the occurrence of something (whether in the

mind or brain). Experience is something we do and its qualitative

features are aspects of this activity. ...“

Explanatory Gap

• „...Another way to put this point is say that qualia-based accounts of

the phenomenology of experience actually misdescribe the

phenomenological character of experience.“ (O‘Regan & Noe, 2001)

• Category mistake.

• No reduction of phenomenal states to physical states. Instead ???

• „lawful relation of dependence between visual stimulation and what

we do, and this lawful relation is determined by the character of the

visual apparatus“ (O‘Regan & Noe, 2001)

• Is this an enterprise of a hermeneutic mutual information between

phenomenology and brain/behaviour science?

What is it like to ...?

• The experience of driving a Porsche:

– no defining sensations

– „one’s comfortable exercise of one’s knowledge of the sensorimotor

contingencies governing the behavior of the car“

• Claim: Vision basically the same

• „We have proposed that experience is a temporally extended

activity of exploration as mediated by the perceiver’s knowledge of

sensorimotor contingencies. The differences in the qualitative

character of perceptual experiences correspond to differences in the

character of the relevant sensorimotor contingencies.“

What is a sensorimotor contingency?

• Rules of sensorimotor contingency = rules of interdependence

between stimulation and movement

• „If you put on inverting lenses, it

is immediately apparent that

eye and head movements

produce surprising patterns,

thus enabling us to direct our

attention to the disruption of

familiar patterns of sensorimotor

contingency.“

Change Blindness

• See change blindness demos on Kevin O’Regan’s webpage:

http://nivea.psycho.univ-paris5.fr/

More demos...

A collection of videos by the Visual Cognition group, University of Illinois:

http://viscog.beckman.uiuc.edu/djs_lab/demos.html

Change blindness demos

• The results of the experiments showed that in many cases

observers have great difficulty seeing changes, even though the

changes are very large, and occur in full view -- they are perfectly

visible to someone who knows what they are.

• In other experiments, O’Regan et al. found that, in many cases,

observers could be looking directly at the change at the moment the

change occurred, and still not see it.

• Change blindness just shows us that details are not always on our

minds.

Closing the Explanatory Gap?

• „You feel the whole bottle. But the differents parts of your hands make

contact only with isolated parts of its surface. Nevertheless, don’t you feel

the whole bottle as present? That is, phenomenologically speaking, the

feeling of presence of the bottle is not a conjecture or an inference. The

feeling you have is the knowledge that movements of the hand open up and

reveal new aspects of bottle surface. It feels to you as if there’s stuff there to

be touched by movement of the hands. That’s what the feeling of the

presence of the bottle consists in. But the basis of the feeling, then, is not

something occurring now. The basis rather is one’s knowledge now as to

what one can do.“ (O‘Regan & Noe, 2001)

• Claim: With sensorimotor contingencies, you can explain qualitative

phenomenal differences between and within perceptual modalities.

What are sensory modalities?

• Most scientists seem satisfied with some variant of

Müller’s [1838] classic concept of “specific nerve

energy”.

• On our view, the differences between the sensory

modalities are to be understood in terms of the different

patterns of sensorimotor contingency governing

perceptual exploration in the different modalities.

Neural Plasticity and Consciousness

• (Hurley & Noe, 2003)

• Cortical Deference vs. Cortical Dominance

– Phantom limbs

– Ferrets

– Braille Reading in blind

To Sum Up:

• Vision not „in brain“

• Vision is an activity of exploration by the whole situated perceiver

• Needs the perceiver’s understanding of what we do (e.g. Eye

movements) leads to what kind of sensation

• „neural activity could [not] be sufficient, as a matter of law, to

produce visual consciousness.“ (O‘Regan & Noe, 2001) („neural

correlate“ research)

• Alternative ways to scientifically explore cognitive behaviour and

capacity scientifically.

Summary of Today’s Lecture

• Situated and Embodied View:

– The closed loop

– The rejection of internal localisation of cognitive phenomena and faculties

– Sensorimotor coordination as reciprocally causal process.

• Empirical research Perceptual perturbations

– Perceptual suppleance (sensory substitution)

– Change blindness

– Delay experiments

• Sensorimotor contingencies

– Descriptive concepts for a situated view.

– Can be used to explain cognitive phenomena and faculties (e.g. perceptual

modalities) without localising meaningful cognitive phenomena

Outlook to Next Lecture

• Computational Modelling of the Closed Loop:– Behaviour Based Robotics– Evolutionary Robotics

• Approaching the hard questions– Autopoiesis and Adaptivity– Values, Emotions and genuine purposes

• Wrapping it all up

Any questions?

References• Angliss, S., Rohde, M. McGregor, S. and W.Bigge: The Cyborg Experiment.

Exhibition and Lecture at the Cheltenham festival of Science June 2005.• Bach-y-Rita, P., M. E. Tyler, and K. A Kaczmarek: Seeing With the Brain. Int. J.

Human-Computer Interaction 15(2) 2003. 285-295.• Cunningham, D.W., A. Chatziastros, M. von der Heyde and H.H. Bülthoff: Driving in

the future: Temporal visuomotor adaptation and generalization. Journal of Vision 1(2), 88-98 (2001)

• Cunningham, D.W., Billock, V.A. and Tsou, B.H.: Sensorimotor adaptation to violations of temporal contiguity. Psychological Science 12(6), 532-535 (2001)

• Di Paolo, E.: lecture presentations “Adaptive Systems”, University of Sussex, Spring Term 2006

• Helbing, D., J. Keltsch and P. Molnar: „Modelling the Evolution of Human Trail Systems“ Nature 388 49, 1997.

• Held, R.: Plasticity in Sensory-Motor Systems. Scientific American, 213 (5) 1965. 84-94.

• Highfield, R.: The Girl With Eyes in the Back of Her Head. Daily Telegraph 5.6.2006.• Hurley, S. and A. Noë, "Neural plasticity and consciousness." Biology and Philosophy

18, 1, pp 131-168 2003• Fujisaki, Shimojo, Kashino and Nishida: Recalibration of audiovisual simultaneity by

adaptation to a constant time lag. J. Vis. 3 (9) Oktober 2003. 34-34.

References• Kohler, I.: Experiments with Goggles. Scientific American, May 1962. • Helbing et al. Nature, 388, pp 45 – 50, (1997). • Held, R.: Plasticity in Sensory-Motor Systems. Scientific American, 213 (5) 1965. 84-94. • Lenay C. (2003) Ignorance et suppléance : la question de l'espace, HDR 2002, Université de

Technologie de Compiègne • Lenay, Charles, Olivier Gapenne, Sylvain Hanneton, Catherine Marque and Christelle Genouëlle

2003. Chapter 16. Sensory substitution: Limits and perspectives. In Touching for Knowing, Hatwell, Yvette, Arlette Streri and Edouard Gentaz (eds.), 275-292.

• Levine, J. "Materialism and Qualia: The Explanatory Gap," Pacific Philosophical Quarterly, vol. 64, no. 4, October, 1983, 354 - 361.

• O’Regan, J.K.’s webpage (retrieved 13.2.2006): http://nivea.psycho.univ-paris5.fr/• O'Regan, J.K. & A. Noë: A sensorimotor account of vision and visual consciousness. in:

Behavioral and Brain Sciences, 2001, 24(5), 939-1011• Smith K.U. and W. M. Smith: Perception and motion: an analysis of space-structured behavior.

Philadelphia, Saunders, 1962.• Stetson, C, X. Cui, P. R. Montague amd D. M. Eagleman: Illusory temporal reversal of action and

effect reveals neural conflict response. Submitted to Neuron. • Swindale, N.V. Brain development: Lightning is always seen, thunder always heard. Current

Biology, 15(2000), 569-571 • Visual Cognition Lab, University of Illinois (retrieved 13.2.2006):

http://viscog.beckman.uiuc.edu/djs_lab/demos.html