Sensory Cue IntegrationMultisensory Predictive Learning, Fall, 2011

Summary by Byoung-Hee KimComputer Science and Engineering (CSE)

http://bi.snu.ac.kr/

¥ Quiz on the gist of the chapter (5 min)¥ Presenters: prepare one main question¥ Students: read the material before the class

¥ Presentation (30 min)¥ Include all equations and figures¥ Limit of slides: maximum 20 pages + appendix (unlimited)

¥ Discussion (30 min)¥ Understanding the contents¥ Pros and cons / benefits and pitfalls¥ Implications of the results¥ Extensions or applications

Multisensory Predictive Learning, Fall, 2011 2

Presentation Guideline

¥ Q. (question on the gist of the chapter) List and explain briefly ideal observer models of cue integration

Multisensory Predictive Learning, Fall, 2011 3

Quiz (5 min)

¥ Motivations and arguments¥ Problems and experiments¥ Ideal-observer models

¥ Linear models for maximum reliability¥ Bayesian estimation and decision making¥ Nonlinear models: generative models and hidden

variables¥ Issues and concerns¥ Appendix

Multisensory Predictive Learning, Fall, 2011 4

Contents

Multisensory Predictive Learning, Fall, 2011 5

Estimation from Various Information

Vision cues

haptic cuesauditory cues

depth

sizelocation

Motion planning

3D orientation

Motor planning

binocular disparity,stereopsis

Texture / shading

Linear perspective

Environment

Sensory information

Cue integration

Estimation and decision/action

Multisensory Predictive Learning, Fall, 2011 6

Uncertain relationship btw cues and environmental properties

- Variability in the mapping btw the cue and a property- Errors in the nervous system’s measurement of the cue- Measured cue values vary unpredictably across viewing conditions and scenes- Estimates may be based on assumptions about the scene and will be flawed if those

assumptions are invalid

Is this optimal?

¥ Studying perceptual computations¥ Modeling cue combination¥ General introduction to the fiend of cue

combination from the perspective of optimal cue integration

Multisensory Predictive Learning, Fall, 2011 7

Motivations

¥ The organism can make more accurate estimates of environmental properties or more beneficial decisions by integrating multiple sources of information

¥ Observers should be more likely to approach optimal behavior in tasks that are important for survival

¥ “ideal-observer” analysis is a critical step in the iterative scientific process of studying perceptual computations

Multisensory Predictive Learning, Fall, 2011 8

Arguments

Multisensory Predictive Learning, Fall, 2011 9

Problems and experiments

EstimationTarget Cues ExperimentalTask

Surface orientation Visual / Haptic

Distance to a drop-off Visual / auditoryWalk blindfolded toward the drop-off / Movement planning

Size Visual / Haptic Checking JND, PSE

Depth Visual (texture, shading)Seeing ridges as real objects or as computer-graphic image

¥ Cue combinations from the perspective of optimal cue integration

¥ Building ideal observers helps formulate the scientific questions that need to be answered before we can understand how the brain solves these problems

¥ Models ¥ Linear models for maximum reliability¥ Bayesian estimation and decision making¥ Nonlinear models: generative models and hidden

variables

Multisensory Predictive Learning, Fall, 2011 10

Ideal-observer models

¥ Assumptions¥ An observer has access to unbiased estimates of a particular world property from each

cue¥ The cues are Gaussian distributed (Gaussian noise) and conditionally independent (n

cues è n independent, Gaussian random variables)

¥ The minimum-variance unbiased estimator is a weighted average of the individual estimates from each cue (eq. 1.1)

Linear models for maximum reliability

Multisensory Predictive Learning, Fall, 2011 11

ri: cue’s reliability (inverse variance )

BAYESIAN ESTIMATION AND DECISION MAKING

Multisensory Predictive Learning, Fall, 2011 12

¥ Pitfalls of the linear model¥ Providing important insights into human perceptual and sensorimotor

processing¥ Only provides a “local” approximation to the ideal observer

¥ Bayes’ Rule

Multisensory Predictive Learning, Fall, 2011 13

Bayesian decision theory as a more general framework

s: scene propertiesd: data

posterior

likelihood prior

Normalizing term

¥ Bayesian decision maker¥ Compute the posterior distribution¥ Choose an estimate, a course of ‘optimal’ action, based on the loss

function¥ An optimal choice of action is one that maximizes expected gain

¥ P(s): A model of the environment. Prior distribution on the scenes¥ P(d|s): Noisy sensory data d conditioned on a particular state of the world¥ a(d): optimal action¥ t: outcome of the decision or action plan. For estimation, ¥ g(t,s): negative of loss, or gain

Multisensory Predictive Learning, Fall, 2011 14

Bayesian decision theory as a more general framework

Special cases- ML estimation- MAP estimation- Mean of the posterior

¥ Cue integration¥ Assumption: sensory data associated with each cue are

conditionally independent¥ Likelihood and posterior

¥ Special cases¥ For Gaussian, the MAP (maximum a posteriori) estimate and the

mean of the posterior both yield a linear estimation procedure¥ Flat prior yield the posterior as the product of cue likelihoods¥ Conditional independence does not hold è weights should

cover the covariance structure of the data

Multisensory Predictive Learning, Fall, 2011 15

Bayesian decision theory and cue integration

¥ Examples of two simple cases

Bayesian integration of sensory cues

• A: Two cues to object size, visual and haptic, each have Gaussian likelihoods

• B: Two visual cues to surface orientation are provided: skew symmetry (a figural cue) and stereo disparity

Multisensory Predictive Learning, Fall, 2011 16

NONLINEAR MODELS: GENERATIVE MODELS AND HIDDEN VARIABLES

Multisensory Predictive Learning, Fall, 2011 17

¥ Conditions under which optimal cue integration is not linear (cues interact)¥ Cue disambiguation

¥ Raw sensory data from different cues are often incommensurate

¥ Mixture priors (Ch. 9)¥ The true prior is a mixture of distributions

¥ Causal inference (Chs. 2, 3, 4, 13)¥ Cues may derive from different sources¥ The observer should infer the structure of the scene before

estimation

Multisensory Predictive Learning, Fall, 2011 18

Problems and models in nonlinear cases

Cue Disambiguation

Multisensory Predictive Learning, Fall, 2011 19

Estimation target

Relative depth Cues

DisparityVelocity

* Promotion: preliminary conversion of cue values into common units

Viewing distance: hidden variable

¥ Discrepant cue: cues may suggest very different values for some scene property

¥ A: compression cue. mixture of likelihood has long tail

¥ B: small cue conflicts. Disparities (red) suggest a slant which is slightly differ from the compression cue (blue)

¥ C: large cue conflicts. Model selection / model switching

Multisensory Predictive Learning, Fall, 2011 20

Use case of a mixture prior:Bayesian model of slant from texture

Long-tail

Estimation target slant Cues

DisparityTexture

¥ Cues may be derived from different sources¥ The observer need to infer the structure of the

scene, not just to estimate¥ Location estimation from auditory and visual

cues¥ When two stimuli are presented in nearby locations,

subjects’ estimates of the auditory stimulus are pulled toward the visual stimulus (the ventriloquist effect)

¥ When they are presented far apart, they appear to be separate sources and do not affect one another

¥ Model: Bayesian inference of structural models¥ Probabilistic description of a generative model of the

scene (two step process in Fig. 1.4)¥ An observer has to invert the generative model and

infer the locations of the visual and auditory sources

Multisensory Predictive Learning, Fall, 2011 21

Causal inference

location

¥ Bayesian decision theory provides a completely general normative framework for cue integration

¥ The representational framework used to model specific problems depends critically on the structure of the information available and the observer’s task

Multisensory Predictive Learning, Fall, 2011 22

Take home messages

THEORY MEETS DATA

Multisensory Predictive Learning, Fall, 2011 23

¥ A variety of experimental techniques has been used to test theories of cue integration

¥ Example: combination of visual and haptic cues to size¥ Four kinds of stimuli: visual-only; haptic-only;

two-cue, consistent stimuli; two-cue inconsistent stimuli

¥ Threshold value (just-noticeable difference, JND) is used to estimate the underlying single-cue noise

¥ To find the point of subjective equality (PSE)

Multisensory Predictive Learning, Fall, 2011 24

Methodology

¥ Experimental supports optimality of human perception¥ Optimal linear cue integration

¥ Cue promotion is an issue for many cue-integration problems

¥ Evidence for robustness in intrasensory cue combination

¥ Human performance appears to be consistent with the predictions of mixture-prior model

Multisensory Predictive Learning, Fall, 2011 25

Overview of results

ISSUES AND CONCERNS

Multisensory Predictive Learning, Fall, 2011 26

¥ Realism and unmodeled cues¥ The lack of realism and the dearth of sensory cues in the

laboratory may place the perceiver in situations for which the nervous systems is ill suited and therefore may perform suboptimally

¥ Considering unmodeled cues seems to be important (Buckley and Frisby, 1993)

¥ Estimation of uncertainty¥ Measurement of the reliability of individual cues¥ For intramodal cue integration, difficulties arise in isolating a cue¥ Single-cue discrimination experiments are used to estimate the

uncertainty associated with individual cues

Multisensory Predictive Learning, Fall, 2011 27

Issues and Concerns

¥ Estimator bias¥ Sensory calibration: sensory estimators maintain

internal consistency and external accuracy¥ Variable cue weights

¥ How human observers estimate and represent cue reliability? One suggestion: neural population code (Ch. 21)

¥ Simulation of the observer¥ Where the prior comes from and how to estimate itè three different approaches in recent years (one in Ch. 11)

Multisensory Predictive Learning, Fall, 2011 28

Issues and Concerns

APPENDIX

Multisensory Predictive Learning, Fall, 2011 29

¥ How is cue reliability estimated and represented in the nervous system?

¥ How optimal is cue integration w.r.t. the information that is available in the environment?

¥ When human cue integration is demonstrably suboptimal, what design considerations does the suboptimality reflect?

Multisensory Predictive Learning, Fall, 2011 30

OPEN QUESTIONS

¥ Computational level¥ what does the system do (e.g.: what problems does it solve

or overcome) and, equally importantly, why does it do these things

¥ Algorithmic/representational level¥ how does the system do what it does, specifically, what

representations does it use and what processes does it employ to build and manipulate the representations

¥ Implementational level¥ how is the system physically realized (in the case of

biological vision, what neural structures and neuronal activities implement the visual system)

Multisensory Predictive Learning, Fall, 2011 31

Marr's Tri-Level Hypothesis

Multisensory Predictive Learning, Fall, 2011 32

Multisensory Predictive Learning, Fall, 2011 33

¥ The real-ridge experiment

¥ The computer-display experiment

Multisensory Predictive Learning, Fall, 2011 34

The real-ridge experiment

Multisensory Predictive Learning, Fall, 2011 35