Post on 28-Jan-2021
Sensory Cue IntegrationMultisensory Predictive Learning, Fall, 2011
Summary by Byoung-Hee KimComputer Science and Engineering (CSE)
http://bi.snu.ac.kr/
¥ Quiz on the gist of the chapter (5 min)¥ Presenters: prepare one main question¥ Students: read the material before the class
¥ Presentation (30 min)¥ Include all equations and figures¥ Limit of slides: maximum 20 pages + appendix (unlimited)
¥ Discussion (30 min)¥ Understanding the contents¥ Pros and cons / benefits and pitfalls¥ Implications of the results¥ Extensions or applications
Multisensory Predictive Learning, Fall, 2011 2
Presentation Guideline
¥ Q. (question on the gist of the chapter) List and explain briefly ideal observer models of cue integration
Multisensory Predictive Learning, Fall, 2011 3
Quiz (5 min)
¥ Motivations and arguments¥ Problems and experiments¥ Ideal-observer models
¥ Linear models for maximum reliability¥ Bayesian estimation and decision making¥ Nonlinear models: generative models and hidden
variables¥ Issues and concerns¥ Appendix
Multisensory Predictive Learning, Fall, 2011 4
Contents
Multisensory Predictive Learning, Fall, 2011 5
Estimation from Various Information
Vision cues
haptic cuesauditory cues
depth
sizelocation
Motion planning
3D orientation
Motor planning
binocular disparity,stereopsis
Texture / shading
Linear perspective
Environment
Sensory information
Cue integration
Estimation and decision/action
Multisensory Predictive Learning, Fall, 2011 6
Uncertain relationship btw cues and environmental properties
- Variability in the mapping btw the cue and a property- Errors in the nervous system’s measurement of the cue- Measured cue values vary unpredictably across viewing conditions and scenes- Estimates may be based on assumptions about the scene and will be flawed if those
assumptions are invalid
Is this optimal?
¥ Studying perceptual computations¥ Modeling cue combination¥ General introduction to the fiend of cue
combination from the perspective of optimal cue integration
Multisensory Predictive Learning, Fall, 2011 7
Motivations
¥ The organism can make more accurate estimates of environmental properties or more beneficial decisions by integrating multiple sources of information
¥ Observers should be more likely to approach optimal behavior in tasks that are important for survival
¥ “ideal-observer” analysis is a critical step in the iterative scientific process of studying perceptual computations
Multisensory Predictive Learning, Fall, 2011 8
Arguments
Multisensory Predictive Learning, Fall, 2011 9
Problems and experiments
EstimationTarget Cues ExperimentalTask
Surface orientation Visual / Haptic
Distance to a drop-off Visual / auditoryWalk blindfolded toward the drop-off / Movement planning
Size Visual / Haptic Checking JND, PSE
Depth Visual (texture, shading)Seeing ridges as real objects or as computer-graphic image
¥ Cue combinations from the perspective of optimal cue integration
¥ Building ideal observers helps formulate the scientific questions that need to be answered before we can understand how the brain solves these problems
¥ Models ¥ Linear models for maximum reliability¥ Bayesian estimation and decision making¥ Nonlinear models: generative models and hidden
variables
Multisensory Predictive Learning, Fall, 2011 10
Ideal-observer models
¥ Assumptions¥ An observer has access to unbiased estimates of a particular world property from each
cue¥ The cues are Gaussian distributed (Gaussian noise) and conditionally independent (n
cues è n independent, Gaussian random variables)
¥ The minimum-variance unbiased estimator is a weighted average of the individual estimates from each cue (eq. 1.1)
Linear models for maximum reliability
Multisensory Predictive Learning, Fall, 2011 11
ri: cue’s reliability (inverse variance )
BAYESIAN ESTIMATION AND DECISION MAKING
Multisensory Predictive Learning, Fall, 2011 12
¥ Pitfalls of the linear model¥ Providing important insights into human perceptual and sensorimotor
processing¥ Only provides a “local” approximation to the ideal observer
¥ Bayes’ Rule
Multisensory Predictive Learning, Fall, 2011 13
Bayesian decision theory as a more general framework
s: scene propertiesd: data
posterior
likelihood prior
Normalizing term
¥ Bayesian decision maker¥ Compute the posterior distribution¥ Choose an estimate, a course of ‘optimal’ action, based on the loss
function¥ An optimal choice of action is one that maximizes expected gain
¥ P(s): A model of the environment. Prior distribution on the scenes¥ P(d|s): Noisy sensory data d conditioned on a particular state of the world¥ a(d): optimal action¥ t: outcome of the decision or action plan. For estimation, ¥ g(t,s): negative of loss, or gain
Multisensory Predictive Learning, Fall, 2011 14
Bayesian decision theory as a more general framework
Special cases- ML estimation- MAP estimation- Mean of the posterior
¥ Cue integration¥ Assumption: sensory data associated with each cue are
conditionally independent¥ Likelihood and posterior
¥ Special cases¥ For Gaussian, the MAP (maximum a posteriori) estimate and the
mean of the posterior both yield a linear estimation procedure¥ Flat prior yield the posterior as the product of cue likelihoods¥ Conditional independence does not hold è weights should
cover the covariance structure of the data
Multisensory Predictive Learning, Fall, 2011 15
Bayesian decision theory and cue integration
¥ Examples of two simple cases
Bayesian integration of sensory cues
• A: Two cues to object size, visual and haptic, each have Gaussian likelihoods
• B: Two visual cues to surface orientation are provided: skew symmetry (a figural cue) and stereo disparity
Multisensory Predictive Learning, Fall, 2011 16
NONLINEAR MODELS: GENERATIVE MODELS AND HIDDEN VARIABLES
Multisensory Predictive Learning, Fall, 2011 17
¥ Conditions under which optimal cue integration is not linear (cues interact)¥ Cue disambiguation
¥ Raw sensory data from different cues are often incommensurate
¥ Mixture priors (Ch. 9)¥ The true prior is a mixture of distributions
¥ Causal inference (Chs. 2, 3, 4, 13)¥ Cues may derive from different sources¥ The observer should infer the structure of the scene before
estimation
Multisensory Predictive Learning, Fall, 2011 18
Problems and models in nonlinear cases
Cue Disambiguation
Multisensory Predictive Learning, Fall, 2011 19
Estimation target
Relative depth Cues
DisparityVelocity
* Promotion: preliminary conversion of cue values into common units
Viewing distance: hidden variable
¥ Discrepant cue: cues may suggest very different values for some scene property
¥ A: compression cue. mixture of likelihood has long tail
¥ B: small cue conflicts. Disparities (red) suggest a slant which is slightly differ from the compression cue (blue)
¥ C: large cue conflicts. Model selection / model switching
Multisensory Predictive Learning, Fall, 2011 20
Use case of a mixture prior:Bayesian model of slant from texture
Long-tail
Estimation target slant Cues
DisparityTexture
¥ Cues may be derived from different sources¥ The observer need to infer the structure of the
scene, not just to estimate¥ Location estimation from auditory and visual
cues¥ When two stimuli are presented in nearby locations,
subjects’ estimates of the auditory stimulus are pulled toward the visual stimulus (the ventriloquist effect)
¥ When they are presented far apart, they appear to be separate sources and do not affect one another
¥ Model: Bayesian inference of structural models¥ Probabilistic description of a generative model of the
scene (two step process in Fig. 1.4)¥ An observer has to invert the generative model and
infer the locations of the visual and auditory sources
Multisensory Predictive Learning, Fall, 2011 21
Causal inference
location
¥ Bayesian decision theory provides a completely general normative framework for cue integration
¥ The representational framework used to model specific problems depends critically on the structure of the information available and the observer’s task
Multisensory Predictive Learning, Fall, 2011 22
Take home messages
THEORY MEETS DATA
Multisensory Predictive Learning, Fall, 2011 23
¥ A variety of experimental techniques has been used to test theories of cue integration
¥ Example: combination of visual and haptic cues to size¥ Four kinds of stimuli: visual-only; haptic-only;
two-cue, consistent stimuli; two-cue inconsistent stimuli
¥ Threshold value (just-noticeable difference, JND) is used to estimate the underlying single-cue noise
¥ To find the point of subjective equality (PSE)
Multisensory Predictive Learning, Fall, 2011 24
Methodology
¥ Experimental supports optimality of human perception¥ Optimal linear cue integration
¥ Cue promotion is an issue for many cue-integration problems
¥ Evidence for robustness in intrasensory cue combination
¥ Human performance appears to be consistent with the predictions of mixture-prior model
Multisensory Predictive Learning, Fall, 2011 25
Overview of results
ISSUES AND CONCERNS
Multisensory Predictive Learning, Fall, 2011 26
¥ Realism and unmodeled cues¥ The lack of realism and the dearth of sensory cues in the
laboratory may place the perceiver in situations for which the nervous systems is ill suited and therefore may perform suboptimally
¥ Considering unmodeled cues seems to be important (Buckley and Frisby, 1993)
¥ Estimation of uncertainty¥ Measurement of the reliability of individual cues¥ For intramodal cue integration, difficulties arise in isolating a cue¥ Single-cue discrimination experiments are used to estimate the
uncertainty associated with individual cues
Multisensory Predictive Learning, Fall, 2011 27
Issues and Concerns
¥ Estimator bias¥ Sensory calibration: sensory estimators maintain
internal consistency and external accuracy¥ Variable cue weights
¥ How human observers estimate and represent cue reliability? One suggestion: neural population code (Ch. 21)
¥ Simulation of the observer¥ Where the prior comes from and how to estimate itè three different approaches in recent years (one in Ch. 11)
Multisensory Predictive Learning, Fall, 2011 28
Issues and Concerns
APPENDIX
Multisensory Predictive Learning, Fall, 2011 29
¥ How is cue reliability estimated and represented in the nervous system?
¥ How optimal is cue integration w.r.t. the information that is available in the environment?
¥ When human cue integration is demonstrably suboptimal, what design considerations does the suboptimality reflect?
Multisensory Predictive Learning, Fall, 2011 30
OPEN QUESTIONS
¥ Computational level¥ what does the system do (e.g.: what problems does it solve
or overcome) and, equally importantly, why does it do these things
¥ Algorithmic/representational level¥ how does the system do what it does, specifically, what
representations does it use and what processes does it employ to build and manipulate the representations
¥ Implementational level¥ how is the system physically realized (in the case of
biological vision, what neural structures and neuronal activities implement the visual system)
Multisensory Predictive Learning, Fall, 2011 31
Marr's Tri-Level Hypothesis
Multisensory Predictive Learning, Fall, 2011 32
Multisensory Predictive Learning, Fall, 2011 33
¥ The real-ridge experiment
¥ The computer-display experiment
Multisensory Predictive Learning, Fall, 2011 34
The real-ridge experiment
Multisensory Predictive Learning, Fall, 2011 35