Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial -...
-
Upload
gavin-garrett -
Category
Documents
-
view
227 -
download
8
Transcript of Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial -...
![Page 1: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/1.jpg)
Neural Networks
![Page 2: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/2.jpg)
Background
- Neural Networks can be :
- Biological models
- Artificial models
- Desire to produce artificial systems capable of sophisticated computations similar to the human brain.
![Page 3: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/3.jpg)
Biological analogy and some main ideas
• The brain is composed of a mass of interconnected neurons– each neuron is connected to many other neurons
• Neurons transmit signals to each other• Whether a signal is transmitted is an all-or-nothing event
(the electrical potential in the cell body of the neuron is thresholded)
• Whether a signal is sent, depends on the strength of the bond (synapse) between two neurons
![Page 4: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/4.jpg)
How Does the Brain Work ? (1)
NEURON
- The cell that performs information processing in the brain.- Fundamental functional unit of all nervous system tissue.
![Page 5: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/5.jpg)
Each consists of :SOMA, DENDRITES, AXON, and SYNAPSE.
How Does the Brain Work ? (2)
![Page 6: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/6.jpg)
Brain vs. Digital Computers (1)
- Computers require hundreds of cycles to simulate a firing of a neuron.
- The brain can fire all the neurons in a single step. Parallelism
- Serial computers require billions of cycles to perform some tasks but the brain takes less than a second.
e.g. Face Recognition
![Page 7: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/7.jpg)
Definition of Neural Network
A Neural Network is a system composed of
many simple processing elements operating in
parallel which can acquire, store, and utilize
experiential knowledge.
![Page 8: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/8.jpg)
Artificial Neural Network?
Neurons vs. Units (1)• Each element of NN is a node called unit.• Units are connected by links.• Each link has a numeric weight.
![Page 9: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/9.jpg)
Neurons vs. units (2)Real neuron is far away from our simplified model - unit
Chemistry, biochemistry, quantumness.
![Page 10: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/10.jpg)
Computing ElementsA typical unit:
![Page 11: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/11.jpg)
Planning in building a Neural Network
Decisions must be taken on the following:
- The number of units to use.- The type of units required.- Connection between the units.
![Page 12: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/12.jpg)
How NN learns a task.
Issues to be discussed
- Initializing the weights.- Use of a learning algorithm.- Set of training examples.- Encode the examples as inputs.- Convert output into meaningful results.
![Page 13: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/13.jpg)
Neural Network Example
A very simple, two-layer, feed-forward network with two inputs, two hidden nodes, and one output node.
![Page 14: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/14.jpg)
Simple Computations in this network
- There are 2 types of components: Linear and Non-linear.- Linear: Input function
- calculate weighted sum of all inputs.- Non-linear: Activation function
- transform sum into activation level.
![Page 15: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/15.jpg)
CalculationsInput function:
Activation function g:
![Page 16: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/16.jpg)
A Computing Unit. Now in more detail but for a particular model only
A unit
![Page 17: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/17.jpg)
Activation Functions
- Use different functions to obtain different models.- 3 most common choices :
1) Step function2) Sign function3) Sigmoid function
- An output of 1 represents firing of a neuron down the axon.
![Page 18: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/18.jpg)
Step Function Perceptrons
![Page 19: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/19.jpg)
3 Activation Functions
![Page 20: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/20.jpg)
Standard structure of an artificial neural network
• Input units– represents the input as a fixed-length vector of numbers (user
defined)
• Hidden units– calculate thresholded weighted sums of the inputs– represent intermediate calculations that the network learns
• Output units– represent the output as a fixed length vector of numbers
![Page 21: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/21.jpg)
Representations• Logic rules
– If color = red ^ shape = square then +• Decision trees
– tree• Nearest neighbor
– training examples• Probabilities
– table of probabilities• Neural networks
– inputs in [0, 1]
Can be used for all of them
Many variants exist
![Page 22: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/22.jpg)
Notation
![Page 23: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/23.jpg)
Notation (cont.)
![Page 24: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/24.jpg)
Operation of individual units
• Outputi = f(Wi,j * Inputj + Wi,k * Inputk + Wi,l * Inputl)– where f(x) is a threshold (activation) function– f(x) = 1 / (1 + e-Output)
• “sigmoid”
– f(x) = step function
![Page 25: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/25.jpg)
Artificial Neural Networks
![Page 26: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/26.jpg)
Perceptron Learning Theorem
• Recap: A perceptron (threshold unit) can learn anything that it can represent (i.e. anything separable with a hyperplane)
26
![Page 27: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/27.jpg)
The Exclusive OR problem
A Perceptron cannot represent Exclusive OR since it is not linearly separable.
27
![Page 28: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/28.jpg)
28
![Page 29: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/29.jpg)
Properties of architecture
• No connections within a layer• No direct connections between input and output layers• Fully connected between layers• Often more than 3 layers• Number of output units need not equal number of input units• Number of hidden units per layer can be more or less than input or output units
y f w x bi ij j ij
m
( )1
Each unit is a perceptron
Often include bias as an extra weight
29
![Page 30: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/30.jpg)
Conceptually: Forward Activity -Backward Error
30
![Page 31: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/31.jpg)
Backward pass phase: computes ‘error signal’, propagates the error backwards through network starting at output units (where the error is the difference between actual and desired output values)
Forward pass phase: computes ‘functional signal’, feed forward propagation of input pattern signals through network
Backpropagation learning algorithm ‘BP’
Solution to credit assignment problem in MLP. Rumelhart, Hinton and Williams (1986) (though actually invented earlier in a PhD thesis relating to economics)
BP has two phases:
31
![Page 32: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/32.jpg)
Forward Propagation of Activity• Step 1: Initialize weights at random, choose a
learning rate η • Until network is trained:• For each training example i.e. input pattern and
target output(s):• Step 2: Do forward pass through net (with fixed
weights) to produce output(s)– i.e., in Forward Direction, layer by layer:
• Inputs applied• Multiplied by weights• Summed• ‘Squashed’ by sigmoid activation function• Output passed to each neuron in next layer
– Repeat above until network output(s) produced
32
![Page 33: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/33.jpg)
Step 3. Back-propagation of error• Compute error (delta or local gradient) for each output unit δ k
• Layer-by-layer, compute error (delta or local gradient) for each hidden unit δ j by backpropagating errors (as shown previously) Step 4: Next, update all the weights Δwij By gradient descent, and go back to Step 2
The overall MLP learning algorithm, involving forward pass and backpropagation of error (until the network training completion), is known as the Generalised Delta Rule (GDR), or more commonly, the Back Propagation (BP) algorithm
33
![Page 34: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/34.jpg)
‘Back-prop’ algorithm summary (with Maths!)
34
![Page 35: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/35.jpg)
‘Back-prop’ algorithm summary (with NO Maths!)
35
![Page 36: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/36.jpg)
MLP/BP: A worked example
36
![Page 37: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/37.jpg)
Worked example: Forward Pass
37
![Page 38: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/38.jpg)
Worked example: Forward Pass
38
![Page 39: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/39.jpg)
Worked example: Backward Pass
39
![Page 40: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/40.jpg)
Worked example: Update WeightsUsing Generalized Delta Rule (BP)
40
![Page 41: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/41.jpg)
Similarly for the all weights wij:
41
![Page 42: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/42.jpg)
Verification that it works
42
![Page 43: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/43.jpg)
Training
• This was a single iteration of back-prop• Training requires many iterations with many
training examples or epochs (one epoch is entire presentation of complete training set)
• It can be slow !• Note that computation in MLP is local (with
respect to each neuron)• Parallel computation implementation is also
possible
43
![Page 44: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/44.jpg)
Training and testing data
• How many examples ?– The more the merrier !
• Disjoint training and testing data sets– learn from training data but evaluate
performance (generalization ability) on unseen test data
• Aim: minimize error on test data
44
![Page 45: Neural Networks. Background - Neural Networks can be : Biological - Biological models Artificial - Artificial models - Desire to produce artificial systems.](https://reader034.fdocuments.us/reader034/viewer/2022042821/56649d8e5503460f94a771b4/html5/thumbnails/45.jpg)
More resources
• Binary Logic Unit in an example – http://www.cs.usyd.edu.au/~irena/ai01/nn/5.html
• MultiLayer Perceptron Learning Algorithm – http://www.cs.usyd.edu.au/~irena/ai01/nn/8.html
45