The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State...
-
Upload
bridget-rodgers -
Category
Documents
-
view
222 -
download
0
Transcript of The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State...
The Time Dimension for Scene Analysis
DeLiang Wang
Perception & Neurodynamics LabThe Ohio State University, USA
Presentation outline
Introduction Scene analysis and temporal correlation theory
Oscillatory Correlation LEGION network
Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem
Concluding remarks
Scene analysis problem
Binding problem
Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons)
Binding problemin Rosenblatt’sperceptrons
Temporal correlation theory
Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74)
Application to cocktail party processing (von der Malsburg & Schneider’86)
Physiological evidence (Gray et al.’89)
Oscillatory correlation theory
• Oscillators represent feature detectors
• Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies
Computational requirements Need to synchronize locally coupled oscillator
population Need to desynchronize different populations, when
facing multiple objects Synchrony and desynchrony
must be achieved rapidly
LEGION architecture
LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)
Relaxation oscillator as building block
Typical x trace (membrane potential)
With stimulus
Without stimulus
Analytical results
Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate
Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained
Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks
Connectedness problem
Minsky-Papert connectedness problem is a long-standing problem in perceptron learning
The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations
Connectedness problem: LEGION solution Basic idea:
Synchronization within a connected pattern and desynchronization between different ones
Presentation outline
Introduction Scene analysis and temporal correlation theory
Oscillatory Correlation LEGION network
Oscillatory Correlation Approach to Scene Analysis Image segmentation Object selection Cocktail party problem
Concluding remarks
Oscillatory correlation approach to scene segmentation
Feature extraction first takes place An visual feature can be pixel intensity, depth, local image patch,
texture element, optic flow, etc. An auditory feature can be a pure tone, amplitude and frequency
modulation, onset, harmonicity, etc.
Connection weights between neighboring oscillators are set to be proportional to feature similarity
Global inhibitor controls granularity of segmentation Larger inhibition results in more and smaller regions
Segments pop out from LEGION in time
Image segmentation example: Demo
Input image
Image segmentation example
Input image Segmentation result
Object selection The slow inhibitor keeps
trace of each pattern, which can be overcome by only more salient (larger) patterns
Unlike traditional winner-take-all dynamics, selection (competition) takes place at the object level Consistent with object-
based attention theory Binding precedes attention,
rather than attention precedes binding (Treisman & Gelade’80)
Results of object selection
Input image LEGION output Selection outputInput LEGION segmentation Selection
Cocktail party problem
• In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem Popularly known as cocktail-party problem (Cherry’53); also
ball-room problem (Helmholtz, 1863)
• Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90)
Auditory scene analysis (ASA) takes place in two conceptual stages: Segmentation. Decompose the acoustic signal into ‘sensory
elements’ (segments) Grouping. Combine segments into groups, so that segments in
the same group likely originate from the same sound source
NeuralOscillatorNetwork
Correlogram
Cross-channelCorrelation
ResynthesisHairCells
CochlearFiltering
Speechand Noise
ResynthesizedSpeech
ResynthesizedNoise
Correlogram (detail)
TimeLag
Neural Oscillator Network (detail)
GroupingLayer
SegmentationLayer
GlobalInhibitor
TimeFrequency
Oscillatory correlation for ASA (Wang & Brown’99)
Fre
quen
cy
Auditory periphery: Cochleagram
Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
Grouping layer: Example
Two streams emerge from the group layer Foreground: left (original mixture ) Background: right
More recent results (Hu & Wang’04):
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
Time (seconds)0.0 1.5
5000
2741
1457
729
315
80
Back to physiology
Chattering cells recorded by Gray & McCormick’96
Burst oscillations are best modeled by relaxation oscillators
Versatility and time dimension
The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) It characterizes artificial intelligence
The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) It characterizes natural intelligence
Time dimension is necessary for versatility Flexible and infinitely extensible Irreplaceable by spatial organization
Conclusion
Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory
Major progress is made towards solving the scene analysis problem
From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis