Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural...
Transcript of Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural...
![Page 1: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/1.jpg)
Natural Image Statistics and Neural Representation
Eero P SimoncelliCenter for Neural Science
New York University
Bruno A OlshusenCenter for Neuroscience
University of California, Davis
PresenterShilin Zhu
Visual Computing CenterUniversity of California San Diego
![Page 2: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/2.jpg)
Introduction
![Page 3: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/3.jpg)
Evolution of Neural System
• Depends on:
• The tasks that the organism must perform
• The computational capabilities and limitations of neurons
• The environment in which the organism lives (this paper)
![Page 4: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/4.jpg)
Why Study Natural Scene Statistics?
• Identify sources of stimulus information for performing natural tasks
• Generate hypotheses for visual mechanisms that might exploit stimulus information
• Design experiments to test hypothesized mechanisms
![Page 5: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/5.jpg)
Key Observations• Sensory neurons are adapted to statistical properties of signals exposed
• Be able to best process most frequently occurred signals
• Build a statistical prior model of environment
• Coding efficiency can provide link between environmental statistics and neural responses
• An information theory perspective
• Natural images can be used to validate statistical models and coding hypothesis
• Large datasets to test experimentally
![Page 6: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/6.jpg)
Relationship Between Neural Processing and Environmental Statistics
• The relationship can derive new computational models based on environmental statistics
• But surprisingly difficult to make link quantitatively precise
• Several hypothesis
• The goal of perception is to produce efficient representation of signal
• Early sensory neurons remove statistical redundancy of input (efficient coding)
![Page 7: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/7.jpg)
Which Environment Shapes the System?
• Specify probability distribution over the space of input signals
• Specify a timescale over which environment shapes the system
• Specify which neurons are meant to satisfy efficiency criterion
![Page 8: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/8.jpg)
Testing Methodologies
• Direct Approach
• Examine the statistics of neural responses under natural stimulation
• Model-Based Approach
• Derive a model for early sensory processing by optimization to provide good description of neural responses
![Page 9: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/9.jpg)
Basic Concepts
![Page 10: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/10.jpg)
A Theory for Computing with Signals• Combinatorial explosion in number of neurons to uniquely represent each visual pattern
• Informational / Coding efficiency is a constraint on neural processing
• Neurons should encode as much information as possible given available computing resources
• Depends on both transformation and input statistics
• The characteristics of simplistic efficient coding criterion
• No mention of noise
• No mention of uncertainty of neural responses to identical stimuli
• Not compression
![Page 11: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/11.jpg)
Efficient Coding in Single Neurons• Activity of single neuron in response to natural environment
• Scalar value: membrane potential, firing rate, …
• Responses have constraints
• Otherwise information is unbounded
• The information-maximizing response distribution (maximum entropy)
• Fix the maximal value: uniform
• Fix the variance: Gaussian
• Fix the mean: exponential
![Page 12: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/12.jpg)
Efficient Coding in Multiple Neurons
• A set of neurons can jointly encoding information
• Piece of information can be duplicated in more than one neuron
• Neural responses must be statistically independent
• Factorial code
• Conditional probability distribution of a single neuron should be fixed
![Page 13: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/13.jpg)
How to Design such a Sensory System?
• We need to decompose input signals into independent responses
• Consider only linear decomposition and second-order properties
• PCA + Whitening (Variance Equalizer)
![Page 14: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/14.jpg)
PCA Often Fails on Natural Images• The inputs are non-Gaussian
• We need to look at statistical properties of order higher than 2 (beyond covariance)
• Alternative choices
• ICA: maximize higher-order moments like kurtosis
![Page 15: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/15.jpg)
Case Studies: Image Statistics
![Page 16: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/16.jpg)
Some Observations on Natural Images
• They are statistically redundant
• We only see a very small fraction
• Perceptual redundancy experiment: 1.4 bits / pixel
• Redundancies are used for modern compression
![Page 17: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/17.jpg)
Intensity Statistics
• The distribution of light intensities in a visual scene
• Biological evidence
• Contrast-response function of monopolar cell in the fly transforms natural scene to uniform distribution
• Firing rates of spiking neurons in visual cortices of cats and monkeys are exponentially distributed
![Page 18: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/18.jpg)
Color Statistics
• Light has a spectral (wavelength) distribution
• People have shown that natural world can be well-represented by low-dimensional space spanned by cone spectral sensitivities
• 3-D subspace approximation by human cones
![Page 19: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/19.jpg)
Spatial Correlations• Neighboring pixels are strongly correlated in intensity
• The spatial statistics in images are translation and scale invariant
• Spectral power falls with frequency f
• Match well with measurement of compound eye of the fly
• Evidence for decorrelation in early spatial visual processing (subtractive inhibition from neighboring photoreceptors)
![Page 20: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/20.jpg)
Higher-Order Statistics• Efficient coding on cortical
processing
• People derive linear basis functions similar to receptive fields in visual cortex (oriented band-pass filters)
• Non-Gaussianity of natural images
• More work to be done besides decor relation and whitening
![Page 21: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/21.jpg)
Sparseness• Gabor filter has sharp peaks at zero
• Representation corresponding to this density (small amplitude responses) has sparseness
• Maximizing sparsity of representation resembles spatial receptive field of simple cells
• Responses are never actually completely independent: non-linearity exists, need rectified function (e.g., squared)
![Page 22: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/22.jpg)
Space-Time Statistics• A full consideration of image statistics must
include time
• Neurons have important temporal response
• We can estimate spatial-temporal power spectrum by 3-D Fourier transform
• The interdependence between spatial and temporal frequency depends on distribution of object motions
• Filtered image can be described in sparse code: few neurons are active across both space and time
![Page 23: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/23.jpg)
Limitation of Efficient Coding• It does not consider what information should be represented
• It does not consider the task that organisms are doing
• Timescale is not considered: evolution, neural development, short-term adaptation
• Some statistical prior and loss / reward function may need to be considered
• Currently only tested on simple stimuli which is easy to control
![Page 24: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/24.jpg)
Discussion and Conclusion
• These models can be seen as single-stage neural network
• Biological evidence suggests hierarchical organization for more complex aspects of image structure
• The models can be extended to other sensory systems such as auditory system
• The relationship between environmental statistics and sensation is encouraging
![Page 25: Natural Image Statistics and Neural Representationshz338/slides/CSE254_Natural_Imgs.pdf · Natural Image Statistics and Neural Representation Eero P Simoncelli Center for Neural Science](https://reader035.fdocuments.us/reader035/viewer/2022062510/6140eae583382e045471c1c5/html5/thumbnails/25.jpg)
–The most important take away
“The human visual system is the result of evolution by natural selection, and hence its design must incorporate detailed knowledge of the physical regularities of the natural environment. And sparse coding is one important model to describe it.”