Introduction to Neural Networks. Neural Networks in the Brain Human brain “computes” in an...
-
Upload
irma-mosley -
Category
Documents
-
view
222 -
download
5
Transcript of Introduction to Neural Networks. Neural Networks in the Brain Human brain “computes” in an...
Introduction to Neural Networks
Neural Networks in the Brain
• Human brain “computes” in an entirely differ-ent way from conventional digital computers.
• The brain is highly complex, nonlinear, and parallel.
• Organization of neurons to perform tasks much faster than computers. (Typical time taken in visual recognition tasks is 100–200 ms.)
• Key features of the biological brain: experience shapes the wiring through plasticity, and hence learning becomes the central issue in neural networks.
Neural Networks as an Adaptive Machine
• A neural network is a massively parallel distrib-uted processor made up of simple processing units, which has a natural propensity for storing experimental knowledge and making it avail-able for use.
• Neural networks resemble the brain: – Knowledge is acquired from the environment
through a learning process. – Iner neuron connection strengths, known as synaptic
weights, are used to store the acquired knowledge.
• Procedure used for learning: learning algorithm. Weights, or even the topology can be adjusted.
Benefits of Neural Networks
• Nonlinearity: nonlinear components, distributed non-linearity
• Input-output mapping: supervised learning, nonpara-metric statistical inference (model-free estimation, no prior assumptions)
• Adaptivity: either retain or adapt. Can deal with non-stationary environments. Must overcome stability-plas-ticity dilemma.
• Evidential response: decision plus confidence of the decision can be provided.
• Contextual information: Every neuron in the network potentially influences every other neuron, so contex-tual information is dealt with naturally
Benefits of Neural Networks
• Fault tolerance: performance degrades grace-fully.
• VLSI implementability: network of simple com-ponents.
• Uniformity of analysis and design: common components (neurons), sharability of theories and learning algorithms, and seamless integra-tion based on modularity.
• Neurobiological analogy: Neural nets motivated by neurobiology, and neurobiology also turning to neural networks for insights and tools.
Human Brain
• Stimulus → Receptors ⇔ Neural Net ⇔ Effectors → Response: Arbib (1987)
• Neurons are slow: 10−3 s per operation, compared to 10−9 s of modern CPUs.
• Huge number of neurons and on nec-tions: 1010 neurons, 6 × 1013 connec-tions in human brain.
• Highly energy efficient: 10−16J in the brain vs. 10−6 J in modern computers
The Biological Neuron
• Each neuron is a cell that uses biochemical reactions to receive, process and transmit information.
• Each terminal button is connected to other neurons across a small gap called a synapse.
• A neuron's dendritic tree is connected to a thousand neighbouring neurons. When one of those neurons fire, a positive or negative charge is received by one of the dendrites. The strengths of all the received charges are added together through the processes of spatial and temporal summation
Models of Neurons
Inputvalues
weights
Summingfunction
Bias
b
ActivationfunctionInduced
Field
vOutput
y
x1
x2
xm
w2
wm
w
1
)(
Activation Functions
cvb
cvav
if
if )(
otherwise ))/())(((
if
if
)(
cdabcva
dvb
cva
v
)exp(1
1)(
yxvzv
2
2
1exp
2
1)(
vv
Stochastic Model
• Instead of deterministic activation, stochas-tic activation can be done.
• State of neuron +1,-1}• Probability of firing
• Typical choice of where is a pseudo tem-perature.
• When → 0, the neuron becomes determinis-tic.
• Think of computer simulation
Signal Flow Graph
• Nodes and links• Links: synaptic links and activation links.• Incoming edges: summation• Outgoing edges: replication• Architectural graph simplifies the above and
abstracts out internal neuronal function.
Signal Flow Graph Example
• Turn the above into a signal-flow graph
Feedback
• Feedback gives dynamics (temporal aspect), and it is found in almost every part of the nervous system in every animal.
Feedback
Feedback
• : – converge (infinite mem-
ory, fading)
– linearly diverge
– exponentially diverge
Definition of Neural Net-works
• An information processing system that has been developed as a generalization of mathematical models of human cognition or neurobiology, based on the assumptions that– Information processing occurs at many simple ele-
ments called neurons.– Signals are passed between neurons over connection
links.– Each connection link has an associated weight, which
typically multiplies the signal transmitted.– Each neuron applies an activation function (usually
non-linear) to its net input (sum of weighted input sig-nals) to determine its output signal.
Network Architectures
• The connectivity of a neural network is inti-mately linked with the learning algorithm.– Single-layer feedforward networks: one input
layer, one layer of computing units (output layer), acyclic connections.
– Multilayer feedforward networks: one input layer, one (or more)hidden layers, and one output layer. Recurrent networks: feedback loop exists.
– Recurrent networks: feedback loop exists.
• Layers can be fully connected or partially connected.
Similarity Measures
• Similar inputs from similar classes should pro-duce similar representations, leading to classifi-cation into the same category.– Reciprocal of Euclidean distance
==
– Dot product(inner product)==
• The two are related, when =1 ==
Similarity Measure
• When two vectors and are drawn from two distributions:
–Mean vector: –Mahalanobis distance: =– Covariance matrix