SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4...
-
Upload
johnathan-norman -
Category
Documents
-
view
215 -
download
0
Transcript of SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4...
![Page 1: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/1.jpg)
SHANNON LECTURE
LIVING INFORMATION THEORY
Toby Berger
Cornell University
IEEE ISIT – LAUSANNE
4 July 2002
![Page 2: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/2.jpg)
PART III:
NEURAL COALITIONS AND MAIN THEOREM
![Page 3: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/3.jpg)
DEFINITION OF A COALITION OF SENSORY NEURONS
THE AXONS IN A COALITION OF SENSORY NEURONS FORM MANY OF THEIR SYNAPSES WITH OTHER NEURONS IN THE COALITION (INTER-COALITION FEEDBACK). SOMETIMES THE LOCAL CONNECTIVITY IS CLOSE TO 50%, AS OPPOSED TO ONLY 10-7 BRAINWIDE.
THE REMAINDER OF THE SYNAPSES IN WHICH THE AXONS OF A COALITION ARE INVOLVED ARE SPLIT ON AVERAGE ABOUT EQUALLY BETWEEN “LOWER” NEURONS AND “HIGHER” NEURONS. HERE, “LOWER” MEANS CLOSER TO THE SENSORY ORGAN AND “HIGHER” MEANS CLOSER TO THE ‘TOP BRAIN’.
![Page 4: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/4.jpg)
BOTTOM-UP v. TOP-DOWN
INFORMATION TRAVELING ALONG AXONS THAT GO FROM LOWER TO HIGHER IS CALLED BOTTOM-UP, OR FEEDFORWARD, INFORMATION. THAT WHICH TRAVELS ALONG AXONS THAT GO FROM HIGHER TO LOWER IS CALLED TOP-DOWN, OR FEEDBACK, INFORMATION.
![Page 5: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/5.jpg)
Real neural spikes occur in continuous time.
We introduce a time-discrete model with a time step of circa 2 ms (typical spike width). This captures the essence, provided the exact time of occurrence of a neural spike is not significant to an accuracy much greater than 1 ms.
![Page 6: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/6.jpg)
“FIGURE 1” OF NEURAL INFORMATION THEORY
SCHEMATIC OF A COALITION
![Page 7: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/7.jpg)
PSPs
![Page 8: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/8.jpg)
THE COALITION CHANNEL
))1(),(|)((1
))(),...,1(),0(|)(),...,1((
kykxkyn
kP
nxxynyyP
NEXT SLIDE SHOWS THAT A NEURAL COALITION IS A LINEAR (BUT THRESHOLDED) RANDOMIZED DYNAMICAL SYSTEM
![Page 9: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/9.jpg)
COALITION EQUATION
)()()1(()( kimSk
imQ
imWk
iYUk
mY
i
))()()( TklmSk
lmQ
lmWk
lX
l
imW
)(kimQ
)(kimS
where
= Signed weight of synapse (i,m)
Random size of quantal contribution of synapse (i,m) to PSP of coalition member m at time k
= 0 if synapse (i,m) fails at time k, 1 if not. (The fascinating phenomenon of “quantal failure” deserves a full lecture of its own – see Levy and Baxter, JCN 2002 and Proc. ISIT ’02, p. 18.)
)(),(, klmSk
lmQ
lmW analogously defined
)(kmTT Spiking threshold of neuron m at time k
=
))(()( Tkm
PSPUkmY
![Page 10: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/10.jpg)
MAXIMUM INFORMATION RATE HYPOTHESIS
The input (afferent) process {X(k))} that drives a coalition willhave the property that it maximizes the mutual information rate
))(),...,1();(),...,1((1 nYYnXXIn
between itself and the output (efferent) process {Y(k)} that it generates over all processes that lead to the same or smaller energy expenditure in the Y-neurons.
Remark: Energy is expended both in the synapses in receiving and responding to afferent excitation and in the axons both to restore chemical concentrations during refractory periods following action potential generation and, to a lesser extent, to drive spikes down the axonal ‘transmission lines’.
![Page 11: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/11.jpg)
JUSTIFICATIONS FOR THE MAX INFO RATE HYPOTHESIS
• Brain consumes 25% to 50% of metabolic energy under low physical exertion, so its computations should be energy-aware*.
• Inefficient to build and maintain a coalition and then expend more energy than necessary to transfer whatever information about its afferent process it conveys to its efferent process.
• The max info rate hypothesis leads to Markovianness of {X(k),Y(k)} which reduces latency by encapsulating memory.
*L. Sokoloff (1989), “Circulation and energy metabolism of the brain,” in Basic Neurochemistry: Molecular, Cellular and Medical Aspects, 4th ed., G. Siegel et al., Eds.
![Page 12: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/12.jpg)
The Brain as a Markov Chain
MAIN THEOREM:
IF MAXIMUM INFORMATION RATE HYPTOTHESIS IS TRUE, THEN:
• {(Xk,Yk} is a first-order (non-homog) Markov chain
• {Yk} is a first-order (non-homog) Markov chain
• {Xk} is not necessarily Markovian
PROOF: IF TIME PERMITS(This is joint work with Yuzheng Ying with thanks to Claudio Weidmann.)
![Page 13: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/13.jpg)
REMARKS:
• Max info rate hypothesis says the source {X(k)} is robustly “matched” to the channel’s transition matrix, P(y|x).
• If double matching prevails, as we suspect it does, then the failure rate parameterizes the rate-distortion function, and distortion is measured by a Weber-Fechner fidelity criterion of the form
n
kkykykxdn
1)).1(),(),((1
• The Markovianness of the Main Theorem is essential to the brain’s low-latency processing of sensory information. Without it, bottom-up delay would accumulate too fast to allow for the number of coalitions needed to achieve the sophisticated distinctions of which the brain is capable.
![Page 14: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/14.jpg)
From T. S. Lee and M. Nguyen, Dynamics of subjective contour formation in the early visual cortex. PNAS 98(4):1907-1911, 2001.
Temporal Dynamics of a V1 Neuron’s Response to Real and Illusory Contours
![Page 15: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/15.jpg)
• All problems in bio-IT are open, including those I may have implied are solved.
•What is memory? How is it physically stored and accessed?
• Can the max information rate hypothesis be proved by appealing to a least action principal in chemical statistical mechanics? (Perhaps this can be approached via the fact that the solution of multiphase chemical equilibrium problems is obtained by solving for the minimum of the Gibbs/Helmholtz Free Energy which, in turn, is mathematically fully analogous to calculation of a point on a rate-distortion function.)
OPEN PROBLEMS
![Page 16: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/16.jpg)
CLOSING CONJECTURE
The most remarkable forms of information processing – those we call consciousness, intelligence and creativity – all are deeply reflexive. They are products of our self-awareness. It seems reasonable to conjecture that these forms of thinking emerge from the brain’s pervasive and intricate use of feedback.
![Page 17: SHANNON LECTURE LIVING INFORMATION THEORY Toby Berger Cornell University IEEE ISIT – LAUSANNE 4 July 2002.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649cd85503460f949a1293/html5/thumbnails/17.jpg)