Neural Networks (NN)Part 1

1. NN: Basic Ideas2. Computational Principles3. Examples of Neural Computation

• Neural network: An example

1. NN: Basic Ideas

• A neural network has been shown to be a universal Turing machine; it can compute anything that is computable (Siegelmann & Sontag, 1991).

• Further, if equipped with appropriate algorithms, the neural network can be made into an intelligent computing machine that solves problems in finite time.

• Such algorithms have been developed in recent years:– Backpropagation learning rule (1985)– Hebbian learning rule (1949)– Kohonen’s self-organizing feature map (1982)– Hopfield nets (1982)– Boltzman machine (1986)

• Biological neural network (BNN) vs artificial neural network (ANN)

McCulloch-Pitts Networks (1943)

- Model of artificial neurons that computes Boolean logical functions

where Y, X1, X2 take on binary values of 0 or 1, and W1, W2, Q take on continuous values

(e.g.) for W1 = 0.3, W2 = 0.5, W0 = 0.6

X1 X2 Y0 0 0 0 1 01 0 01 1 1

Boolean AND Computation

Y f W X W X Q ( )1 1 2 2

McCulloch & Pitts (1943, Fig. 1)

Knowledge representation in M-P network Hiring Rule #1:

“A job candidate who has either good grade or prior job experience and also gets strong letters and receives positive mark in interview tends to make a desirable employee and therefore should be hired.”

“Knowledge is in the connection weights (and the threshold).”

Learning in M-P network Hiring Rule #1A:

“A job candidate who has either good grade or prior job experience and also gets strong letters or receives positive mark in interview tends to make a desirable employee and therefore should be hired.”

“Learning through weight modification (e.g., Hebb rule).”

Acquisition of new knowledge in M-P network Hiring Rule #2:

“In addition to Rule 1A, a job candidate who has no prior job experience and receives negative mark in interview shouldn’t be hired.”

“Acquisition of new knowledge through creation of new neurons (i.e., synaptogenesis).”

1. Distributed representation

2. Later inhibition

3. Bi-directional interaction

4. Error-correction learning

5. Hebbian learning

2. Computational Principles

1. Distributed Representation (vs. localist representation)

An object is represented by multiple units; the same unit participates in multiple representations:

Why distributed representation?

1. Efficiency Solve the combinatorial explosion problem: With n binary units, 2n different representations possible. (e.g.) How many English words from a combination of 26 alphabet letters?

2, Robustness (fault-tolerance)Loss of one or a few units does not affect representation.

(e.g., holographic image, Fourier transformation).

2. Lateral Inhibition

Selective activation through activation-dependent inhibitory competition (i.e., WTA).

A Z K

3. Bi-directional Interaction(interactive, recurrent connections)

This top-down-and-bottom-up processing computes constraint-optimization, and is generally faster than the uni-directional computation (e.g., word recognition).

Bi-directional interactions in Speech Perception

4. Error-correction Learning(e.g., backpropagation)

dWik =e*Ai*(Tk - Bk)

A three-layer feed forward network with backpropagation learning can approximates any measurable function to any desired degree of accuracy by increasing the number of hidden units (Hornik et al, 1989, 1990; Hecht-Nielsen, 1989).

However, biological plausibility of backpropagation learning is yet to be confirmed.

dWik =e Bk (Ai - Wik)

5. Hebbian Learning(unsupervised/self-organizing)

Hebbian Rule Encodes Correlational Information

Asymptotically,

Wik = P(Ai = ‘fire |Bk =‘fire’)

In other words, the weight Wik stores information about correlation (i.e., co-firing activities) between the input and output units.

Q1: How about encoding the other half of the correlational information, that is,

P(Bk = ‘fire|Ai =‘fire’)

Q2: Discuss implications of an anti-Hebbian learning rule such as

dWik =- e Ai(Bk - Wik)

Biological Plausibility of Hebbian Learning

The neurophysiological plausibility is well documented (e.g., Levy & Stewards, 1980), including sub-cellular mechanisms (NMDA-mediated long-term potentiation (LTP)).

• Noise-tolerant memory• Pattern completion• Content-addressable memory • Language learning• Sensory motor control• Visual perception• Speech perception• Word recognition• …..

3. Examples of Neural Computation

NETtalk

(program that learns to pronounce English words)

Submarine Sonar