Neural Network Society, Japanese Neural Network Society 15 ...
Neural Network 1
-
Upload
shekharincredible -
Category
Documents
-
view
219 -
download
0
Transcript of Neural Network 1
-
8/8/2019 Neural Network 1
1/34
Intro to Neural NetworksIntro to Neural NetworksNiranjan PandaNiranjan Panda
29.10.09
-
8/8/2019 Neural Network 1
2/34
Before we startBefore we start
Information processing technologyInformation processing technologyinspired by studies of brain and theinspired by studies of brain and the
nervous system.nervous system.
-
8/8/2019 Neural Network 1
3/34
What is a Neural Network?What is a Neural Network?
"...a computing system made up of a number of"...a computing system made up of a number of
simple, highly interconnected processingsimple, highly interconnected processing
elements, which process information by theirelements, which process information by their
dynamic state response to external inputs.dynamic state response to external inputs.
-
8/8/2019 Neural Network 1
4/34
Biological NeuronsBiological Neurons
Inputs Outputs
Connection between cells
-
8/8/2019 Neural Network 1
5/34
ContdContd
The brain is made up of neurons which have
A cell body (soma)
Dendrites (inputs)
An axon (outputs)
Synapses (connection between cells)
There are around 1011 neurons, 1014 synapses
are present in human body and they are connected
in massively parallel.
-
8/8/2019 Neural Network 1
6/34
What is a Artificial Neural Network?What is a Artificial Neural Network?
An ANN is a network of many very simpleAn ANN is a network of many very simple
processors ( units ), each possibly having a ( smallprocessors ( units ), each possibly having a ( small
amount of) local memory. The units are connected byamount of) local memory. The units are connected byunidirectional communication channels (connections"),unidirectional communication channels (connections"),
which carry numeric ( as opposed to symbolic ) data.which carry numeric ( as opposed to symbolic ) data.
The units operate only on their local data and on theThe units operate only on their local data and on the
inputs they receive via the connections.inputs they receive via the connections.
-
8/8/2019 Neural Network 1
7/34
HISTORICALHISTORICAL
19431943 ------ McCulloch and Pitts (start of the modern eraMcCulloch and Pitts (start of the modern eraof neural networks).of neural networks). Logical calculus of neuralLogical calculus of neuralnetworks. A network consists of sufficient number ofnetworks. A network consists of sufficient number ofneurons (using a simple model) and properly setneurons (using a simple model) and properly setsynaptic connections can compute any computablesynaptic connections can compute any computable
function.function.
19491949 ------ Hebb's book "The organization ofHebb's book "The organization ofbehavior".behavior". An explicit statement of a physiologicalAn explicit statement of a physiologicallearning rule for synaptic modification was presentedlearning rule for synaptic modification was presented
for the first time.for the first time.
Hebb proposes that the connectivity of the brain isHebb proposes that the connectivity of the brain iscontinually changing as an organism learns differingcontinually changing as an organism learns differingfunctional tasks, and that neural assemblies arefunctional tasks, and that neural assemblies are
created by such changes.created by such changes.
-
8/8/2019 Neural Network 1
8/34
HISTORICALHISTORICAL
19581958 ------ Rosenblatt introduced Perceptron ARosenblatt introduced Perceptron Anovel method of supervised learning. Perceptronnovel method of supervised learning. Perceptronconvergence theorem.Least meanconvergence theorem.Least mean--square (LMS)square (LMS)algorithmalgorithm
19691969 ------ Minsky and Papert showed limits onMinsky and Papert showed limits onperceptron computation. Minsky and Papertperceptron computation. Minsky and Papertshowed that there are fundamental limits onshowed that there are fundamental limits on
what singlewhat single--layer perceptrons can compute.layer perceptrons can compute.
19821982 ------ Hopfield's networks Hopfield showedHopfield's networks Hopfield showedhow to use "Ising spin glass" type of model tohow to use "Ising spin glass" type of model tostore information in dynamically stable networks.store information in dynamically stable networks.
-
8/8/2019 Neural Network 1
9/34
HISTORICALHISTORICAL
19821982 ------ Kohonen's selfKohonen's self--organizing maps (SOM)organizing maps (SOM)Kohonen's selfKohonen's self--organizing maps is capable oforganizing maps is capable ofreproducing important aspects of the structure ofreproducing important aspects of the structure ofbiological neural nets: Data representation usingbiological neural nets: Data representation usingtopographic maps (which are common in the nervoustopographic maps (which are common in the nervous
systems). SOM also has a wide range of applications.systems). SOM also has a wide range of applications.
19851985 ------ Ackley, Hinton, and Sejnowski, developedAckley, Hinton, and Sejnowski, developedBoltzmann machine, which was the first successfulBoltzmann machine, which was the first successful
realization of a multilayer neural network.realization of a multilayer neural network.
19861986 ------ Rumelhart, Hinton, and Williams developedRumelhart, Hinton, and Williams developedthe backthe back--propagation algorithmpropagation algorithm ------ the most popularthe most popularlearning algorithm for the training of multilayerlearning algorithm for the training of multilayer
perceptrons.perceptrons.
-
8/8/2019 Neural Network 1
10/34
Why Neural Networks?Why Neural Networks?
Adaptive learning:Adaptive learning:An ability to learn how to doAn ability to learn how to dotasks based on the data given for training ortasks based on the data given for training orinitial experience.initial experience.
SelfSelf--OrganizationOrganization: An ANN can create its own: An ANN can create its own
organization or representation of the informationorganization or representation of the informationit receives during learning time.it receives during learning time.
Real Time OperationReal Time Operation: ANN computations may: ANN computations maybe carried out in parallel, and special hardwarebe carried out in parallel, and special hardwaredevices are being designed and manufactureddevices are being designed and manufactured
which take advantage of this capability.which take advantage of this capability.Fault Tolerance via Redundant InformationFault Tolerance via Redundant InformationCodingCoding: Partial destruction of a network leads to: Partial destruction of a network leads tothe corresponding degradation of performance.the corresponding degradation of performance.However, some network capabilities may beHowever, some network capabilities may be
retained even with major network damage.retained even with major network damage.
-
8/8/2019 Neural Network 1
11/34
Artificial Neural NetworkArtificial Neural Network
ANNs incorporate the two fundamental components
of biological neural networks:
1. Neurons (nodes)
The basic computational unit.
2. Synapses (weights)
Connection links characterizedby certain weight known as synaptic weight.
-
8/8/2019 Neural Network 1
12/34
Synapse
vs.
weight
Neuron
vs.
Node
-
8/8/2019 Neural Network 1
13/34
Application areaApplication area
AerospaceAerospace
AutomotiveAutomotiveBankingBanking
DefenseDefense
ElectronicsElectronicsEntertainmentEntertainment
Finance &Finance &
InsuranceInsurance
Several areas are there where Neural Network works:Several areas are there where Neural Network works:
ManufacturingManufacturing
MedicalMedical
ApplicationsApplications
Oil and GasOil and Gas
RoboticsRobotics
Speech ProcessingSpeech Processing
TelecommunicationTelecommunication
TransportationTransportation
-
8/8/2019 Neural Network 1
14/34
Example ApplicationsExample Applications
NETtalk (Sejnowski and Rosenberg, 1987)NETtalk (Sejnowski and Rosenberg, 1987)
Maps character strings into phonemes forMaps character strings into phonemes forlearning speech from text.learning speech from text.
Neurogammon (Tesauro and Sejnowski, 1989)Neurogammon (Tesauro and Sejnowski, 1989) Backgammon learning programBackgammon learning program
Speech recognition (Waibel, 1989)Speech recognition (Waibel, 1989)
Converts sound to textConverts sound to text
Character recognition (Le Cun et al., 1989)Character recognition (Le Cun et al., 1989)
Face Recognition (Mitchell)Face Recognition (Mitchell)
ALVINN (Pomerleau, 1988)ALVINN (Pomerleau, 1988)
-
8/8/2019 Neural Network 1
15/34
A Simple Artificial Neuron
Synapticweights
wij
Summing
function
Input
signal
Bias
b
Activation
functionLocal
Field
vOutput
y
x1
xj
xm wim
wi1
/ /
)(N/
-
8/8/2019 Neural Network 1
16/34
Contd..
The neuron is the basic information processing unit of aThe neuron is the basic information processing unit of a
NN. It consists of:NN. It consists of:
11 A set ofA set ofsynapsessynapses ororconnecting linksconnecting links, each link, each link
characterized by acharacterized by a weightweight known asknown as synaptic weightsynaptic weight::WWi1i1 , W, Wi2i2, ,W, ,Wiijj ,, W,, Wimim
22 AnAn adderadderfunction (linear combiner)function (linear combiner)
which computes the weighted sumwhich computes the weighted sum
of the inputs:of the inputs:
33 Activation functionActivation function (squashing function)(squashing function) for limitingfor limiting
the amplitude of the output of the neuron.the amplitude of the output of the neuron.
! !
m
1 jijxwu j
)(uy b!N
-
8/8/2019 Neural Network 1
17/34
Contd..Contd..
BiasBias b (b (weight is w0 but with a fixed input of x0 =1weight is w0 but with a fixed input of x0 =1))has the effect of applying anhas the effect of applying an affine transformationaffine transformation toto uu
i.e.i.e. it serves the purpose of increasing or decreasingit serves the purpose of increasing or decreasingthe net input of activation function depending onthe net input of activation function depending on
whether it is +ve orwhether it is +ve or ve.ve.
v = u + bv = u + b
vvis theis the induced fieldinduced field of the neuronof the neuron
!!
m
1jijxwu
j
v
u
-
8/8/2019 Neural Network 1
18/34
Contd..Contd..
Input
signal
Synaptic
weights
Summing
function
Activation
functionLocal
Field
vOutput
y
x1
x2
xm
w2
wm
w1
/ / )(N
w0x0= +1
bw
xwvm
i
!
! !
0
0
Bias is an external parameter of the neuron.Can be modeled by adding an extra input.
-
8/8/2019 Neural Network 1
19/34
Activation FunctionsActivation Functions
Threshold Function:Threshold Function:
(v) = 1 , v(v) = 1 , v 0 0
= 0 , v < 0= 0 , v < 0
(v) output
1
0 v input
-
8/8/2019 Neural Network 1
20/34
Contd..Contd..
Signum Function:Signum Function:
(v) = +1 , v >(v) = +1 , v >
== --1 , v 1 , v
(v) output
1
0 v input
-1
-
8/8/2019 Neural Network 1
21/34
-
8/8/2019 Neural Network 1
22/34
Contd..Contd..
Sigmoid Function:Sigmoid Function:
(v) = where(v) = where is the slope parameter
= 0.5 , v = 0= 0.5 , v = 0
= 1 , v = = 1 , v =
= 0 , v == 0 , v = --
1
1 + exp ( -v )
-
8/8/2019 Neural Network 1
23/34
Contd..Contd..
Hyperbolic Tangent Function:Hyperbolic Tangent Function:
(v) = tan hv(v) = tan hv
(v) 1 , v infinite(v) 1 , v infinite
--1 , v1 , v --infiniteinfinite
-
8/8/2019 Neural Network 1
24/34
-
8/8/2019 Neural Network 1
25/34
Single Layer Feedforward NetworkSingle Layer Feedforward Network
The nodes on the left are in the soThe nodes on the left are in the so--calledcalled input layer.input layer.
The input layer neurons are to only pass and distribute theThe input layer neurons are to only pass and distribute the
inputs and perform no computation. Thus, the only true layer ofinputs and perform no computation. Thus, the only true layer of
neurons is the one on the right.neurons is the one on the right.
Each of the inputsEach of the inputsx1,x2,xNx1,x2,xN is connected to every artificialis connected to every artificial
neuron in the output layer through the connection weight.neuron in the output layer through the connection weight.Since every value of outputsSince every value of outputs y1,y2,yNy1,y2,yNisis
calculated from the same set ofcalculated from the same set of
input values, eachinput values, each
output is aried basedoutput is aried based
on the connection weights.on the connection weights.
Although the presented network isAlthough the presented network isfully connectedfully connected, the true biological neural, the true biological neural
network may not have all possible connectionsnetwork may not have all possible connections -- the weight value of zero can bethe weight value of zero can be`` "`` "
I
nputlayer Outputlayer
-
8/8/2019 Neural Network 1
26/34
Multilayer Feedforward NetworkMultilayer Feedforward Network
Input
layer
Hidden Layer
Output
layer
To achieve higher level of computational capabilities, a moreTo achieve higher level of computational capabilities, a more
complex structure of neural network is required. Figure showscomplex structure of neural network is required. Figure showsthethe multilayer neural networkmultilayer neural networkwhich distinguishes itself from thewhich distinguishes itself from thesinglesingle--layer network by having one or morelayer network by having one or more hidden layershidden layers. In. Inthis multilayer structure, the input nodes pass the information tothis multilayer structure, the input nodes pass the information tothe units in the first hidden layer, then the outputs from the firstthe units in the first hidden layer, then the outputs from the first
hidden layer are passed to the next layer, and so on.hidden layer are passed to the next layer, and so on.
Multilayer network can be also viewed as cascading of groupsMultilayer network can be also viewed as cascading of groupsof singleof single--layer networks.layer networks.
The level of complexityThe level of complexity
in computing can bein computing can be
Seen by the fact thatSeen by the fact thatmany singlemany single--layerlayer
networks are combinednetworks are combined
into this multilayer network.into this multilayer network.
The designer of an artificialThe designer of an artificial
neural network should consider how many hidden layers areneural network should consider how many hidden layers are
-
8/8/2019 Neural Network 1
27/34
Recurrent NetworksRecurrent Networks
In this type of networks there exists at least 1 feedback loop.In this type of networks there exists at least 1 feedback loop.Output layer is again feed the outputs to the input layer(Output layer is again feed the outputs to the input layer(arbitrary neurons ) as inputs of the same network.arbitrary neurons ) as inputs of the same network.
There could be neurons with self feedback links.There could be neurons with self feedback links.
-
8/8/2019 Neural Network 1
28/34
Learning Algorithm
There are a lot of learning algorithm classified as
supervised learning and unsupervised Learning.
Supervised Learning uses a set of inputs for which
the appropriate (desired) output areknown.Computed output and correct output are
compared to determine error.
Unsupervised Learning only input stimuli are shown
to the network. The network is self-Organizing. Thesystem learns of its own by discovering and adapting
to structural features in input pattern.
-
8/8/2019 Neural Network 1
29/34
2 Main Types of ANN
UnsupervisedSupervised
e.g:
Adaline
Perceptron
MLP
RBF
Fuzzy ARTMAP
etc.
e.g:
Competitive learning networks
- SOM- ART families
- neocognition
- etc.
-
8/8/2019 Neural Network 1
30/34
Supervised Network
Teacher
ANN
+
-
error
-
8/8/2019 Neural Network 1
31/34
Unsupervised ANN
Teacher
ANN
-
8/8/2019 Neural Network 1
32/34
How does an ANN learn
I
N
P
U
T
S
I
G
N
A
L
S
O
U
T
P
U
T
S
I
G
N
A
L
SInput
layer
Middle
layer
Output
Layer
weights
neurons
Connected by links-each link
has a numerical weight
Weight
basic means of long-term
memory in ANNs
Express the strength
Learns through repeated
adjustments of theseweights
-
8/8/2019 Neural Network 1
33/34
Learning Process of ANN
Compute
output
Is
DesiredOutput
achieved
Stop
Adjust
Weight
yes
No
Learn from experience
Learning algorithms
Recognize pattern of
activities
Involves 3 tasks
Compute outputs
Compare outputs with
desired targets
Adjust the weights and
repeat the process
-
8/8/2019 Neural Network 1
34/34
NN Learning Algorithm
Unsupervised Learning Reinforced Learning( Outpt Based )
Supervised Learning( ErrorBased )
ErrorCorrectionGradient descent
HebbianStochastic
Competitive
LeastMean Square
Backpropagation
Learning Algorithm
Classification