3836801 Neural Networks

8/10/2019 3836801 Neural Networks

1/32

NEURAL NETWORKS

Dr. Atif AL-

nsourPresented by:

Abdullah smadi (2004974044)

ohammed bonyan (2004974094)

ustafa al !adi (200497404")


2/32

Abstract:

This report is an introduction to Artificial Neural

Networks. The different types of neural networks are

explained and shown, applications of neural

networks like ANNs in medicine are described, and

historical background is provided. The connectionbetween the artificial and the real thing is also

explained. Finally, the mathematical models

involved are presented.

2


3/32

Contents:

1. Introduction to Neural Networks

1.1 What is a neural network?

1.2 Historical background

1.3 Why use neural networks?

1. Neural networks !ersus con!entional co"#uters $ a co"#arison

2. Hu"an and Arti%icial Neurons $ in!estigating the si"ilarities

2.1 How the Hu"an &rain 'earns?

2.2 (ro" Hu"an Neurons to Arti%icial Neurons

3. An )ngineering a##roach

3.1 A si"#le neuron $ descri#tion o% a si"#le neuron

3.2 (iring rules $ How neurons "ake decisions

3.3 *attern recognition $ an e+a"#le3. A "ore co"#licated neuron

. Architecture o% neural networks

.1 (eed$%orward ,associati!e- networks

.2 (eedback ,auto associati!e- networks

.3 Network layers

. *erce#tions

. /he 'earning *rocess

.1 /rans%er (unction

.2 An )+a"#le to illustrate the abo!e teaching #rocedure

.3 /he &ack$*ro#agation Algorith". /he &ack$*ro#agation Algorith" ,non$linear-

0. A##lications o% neural networks

0.1 Neural networks in #ractice

0.2 Neural networks in "edicine

0.2.1 odeling and iagnosing the Cardio!ascular yste"

0.2.2 )lectronic noses $ detection and reconstruction o% odors by ANNs

0.2.3 Instant *hysician $ a co""ercial neural net diagnostic #rogra"

0.3 Neural networks in business

0.3.1 arketing0.3.2 Credit e!aluation

4. Conclusion

5e%erences

3
http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Introduction%20to%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#What%20is%20a%20Neural%20Networkhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Historical%20backgroundhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Why%20use%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20networks%20versus%20conventional%20computershttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Human%20and%20Artificial%20Neurones%20-%20investigating%20the%20similaritieshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#How%20the%20Human%20Brain%20Learnshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#%20From%20Human%20Neurones%20to%20Artificial%20Neuroneshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#An%20engineering%20approachhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#A%20simple%20neuronhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#%20Firing%20ruleshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Pattern%20Recognition%20-%20an%20examplehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Pattern%20Recognition%20-%20an%20examplehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#A%20more%20complicated%20neuronhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Architecture%20of%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Feed-forward%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Feedback%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Network%20layershttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Perceptronshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#The%20Learning%20Processhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Transfer%20Functionhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#An%20Example%20to%20illustrate%20the%20above%20teaching%20procedurehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#The%20Back-Propagation%20Algorithmhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Applications%20of%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20Networks%20in%20Practicehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20networks%20in%20medicinehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Modelling%20and%20Diagnosing%20the%20Cardiovascular%20Systemhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Electronic%20noseshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Instant%20Physicianhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20Networks%20in%20businesshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Marketinghttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Credit%20Evaluationhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Credit%20Evaluationhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Conclusionhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Referenceshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#What%20is%20a%20Neural%20Networkhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Historical%20backgroundhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Why%20use%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20networks%20versus%20conventional%20computershttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Human%20and%20Artificial%20Neurones%20-%20investigating%20the%20similaritieshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#How%20the%20Human%20Brain%20Learnshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#%20From%20Human%20Neurones%20to%20Artificial%20Neuroneshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#An%20engineering%20approachhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#A%20simple%20neuronhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#%20Firing%20ruleshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Pattern%20Recognition%20-%20an%20examplehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#A%20more%20complicated%20neuronhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Architecture%20of%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Feed-forward%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Feedback%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Network%20layershttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Perceptronshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#The%20Learning%20Processhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Transfer%20Functionhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#An%20Example%20to%20illustrate%20the%20above%20teaching%20procedurehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#The%20Back-Propagation%20Algorithmhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Applications%20of%20neural%20networkshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20Networks%20in%20Practicehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20networks%20in%20medicinehttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Modelling%20and%20Diagnosing%20the%20Cardiovascular%20Systemhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Electronic%20noseshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Instant%20Physicianhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Neural%20Networks%20in%20businesshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Marketinghttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Credit%20Evaluationhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Conclusionhttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Referenceshttp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Introduction%20to%20neural%20networks


4/32

1. Introduction to neural networks

"." %hat is a &eural &et'or

An Artificial Neural Network (ANN is an information processing paradigm that is

inspired by the way biological nervous systems works, such as the brain, process

information. The key element of this paradigm is the structure of the information

processing system. !t is composed of a large number of highly interconnected

processing elements (neurons working together to solve specific problems.

ANNs,its "ust like people, learn by example. An ANN is designed for a specific

application, such as a data classification, through a learning process. #earning in

biological systems involves ad"ustments to the synaptic connections that exist

between the neurons. This is true of ANNs as well.

".2 *istori+al ba+!round

Neural network simulations appear to be a recent development. $owever, this field

was established before the advent of computers, and has survived at least one

ma"or setback and several eras.

%any important advances have been boosted by the use of inexpensive computer

emulations. Following an initial period of enthusiasm, the field survived a period

of frustration and disrepute. &uring this period when funding and professionalsupport was minimal, important advances were made by relatively few researchers.

These pioneers were able to develop convincing technology which surpassed the

limitations identified by %insky and 'apert. %insky and 'apert, published a book

(in )*) in which they summed up a general feeling of frustration (against neural

networks among researchers, and was thus accepted by most without further

analysis. +urrently, the neural network field en"oys a resurgence of interest and a

corresponding increase in funding.

The first artificial neuron was produced in )- by the neurophysiologist arren%c+ulloch and the logician alter 'its. /ut the technology available at that time

did not allow them to do too much.

4


5/32

"., %hy use neural net'ors

Neural networks, with their ability to derive meaning from complicated or

imprecise data, can be used to get patterns and detect trends that are too complex to

be noticed by either humans or other computer techni0ues. A trained neuralnetwork can be thought of as an 1expert1 in the category of information it has been

given to analy2e. This expert can then be used to provide pro"ections given new

situations of interest and answer 1what if1 0uestions.

3ther advantages include4

1. Adaptive learning: An ability to learn how to do tasks basedon the data given for training or initial experience.

2. Self-rgani!ation: An A"" can create its own organi!ation or

representation of the infor#ation it receives d$ring learningti#e.3. %eal &i#e peration: A"" co#p$tations #ay be carried o$t

in parallel' and special hardware devices are being designedand #an$fact$red which take advantage of this capability.

4. (a$lt &olerance via %ed$ndant )nfor#ation *oding: +artialdestr$ction of a network leads to the correspondingdegradation of perfor#ance. ,owever' so#e networkcapabilities #ay be retained even with #aornetworkda#age.

".4 &eural net'ors ersus +onentional +omuters

Neural networks take a different approach to problem solving than that of

conventional computers. +onventional computers use an algorithmic approach i.e.

the computer follows a set of instructions in order to solve a problem. 5nless the

specific steps that the computer needs to follow are known the computer cannot

solve the problem. That restricts the problem solving capability of conventional

computers to problems that we already understand and know how to solve. /ut

computers would be so much more useful if they could do things that we don6t

exactly know how to do.


6/32

Neural networks process information in a similar way the human brain does.

The network is composed of a large number of highly interconnected

processing elements (neurones working in parallel to solve a specificproblem. Neural networks learn by example. They cannot be programmed to

perform a specific task. The examples must be selected carefully otherwise

useful time is wasted or even worse the network might be functioning

incorrectly. The disadvantage is that because the network finds out how tosolve the problem by itself, its operation can be unpredictable.

3n the other hand, conventional computers use a cognitive approach to

problem solving7 the way the problem is to solved must be known and stated

in small unambiguous instructions. These instructions are then converted to a

high level language program and then into machine code that the computer

can understand. These machines are totally predictable7 if anything goes

wrong is due to a software or hardware fault.

Neural networks and conventional algorithmic computers are not in

competition but complement each other. There are tasks are more suited to an

algorithmic approach like arithmetic operations and tasks that are more suited

to neural networks. 8ven more, a large number of tasks, re0uire systems that

use a combination of the two approaches (normally a conventional computer

is used to supervise the neural network in order to perform at maximum

efficiency.

/


7/32

2. Hu"an and Arti%icial Neurons $ in!estigating the

si"ilarities

2." *o' the *uman /rain Learns

%uch is still unknown about how the brain trains itself to process information, so

theories abound. !n the human brain, a typical neuron collects signals from others

through a host of fine structures called dendrites. The neuron sends out signals of

electrical activity through a long, thin stand known as an axon, which splits into

thousands of branches. At the end of each branch, a structure called asynapse

converts the activity from the axon into electrical effects that inhibit or excite

activity in the connected neurons. hen a neuron receives excitatory input that is

sufficiently large compared with its inhibitory input, it sends a spike of electrical

activity down its axon. #earning occurs by changing the effectiveness of the

synapses so that the influence of one neuron on another changes.

+omponents of a neuron The synapse

0


8/32

2.2 rom *uman &eurons to Artifi+ial &eurons

e conduct these neural networks by first trying to find the essential features of

neurones and their interconnections. e then typically program a computer tosimulate these features. $owever because our knowledge of neurones is

incomplete and our computing power is limited, our models are necessarily gross

ideali2ations of real networks of neurones.

/he neuron "odel.


9/32

6 3. An engineering a##roach

,." A simle neuron

An artificial neuron is a device with many inputs and one output. The neuron has

two modes of operation7 the training mode and the using mode. !n the training

mode, the neuron can be trained to fire (or not, for particular input patterns. !n the

using mode, when a taught input pattern is detected at the input, its associated

output becomes the current output. !f the input pattern does not belong in the

taught list of input patterns, the firing rule is used to determine whether to fire or

not.

A simple neuron

,.2 irin! rules

The firing rule is an important concept in neural networks and accounts for their

high flexibility. A firing rule determines how one calculates whether a neuron

should fire for any input pattern. !t relates to all the input patterns, not only the

ones on which the node was trained.

A simple firing rule can be implemented by using $amming distance techni0ue.

The rule goes as follows4

Take a collection of training patterns for a node, some of which cause it to fire (the

9taught set of patterns and others which prevent it from doing so (the :9taught

set. Then the patterns not in the collection cause the node to fire if, on comparison

, they have more input elements in common with the 6nearest6 pattern in the 9

taught set than with the 6nearest6 pattern in the :9taught set. !f there is a tie, then the

pattern remains in the undefined state.


10/32

For example, a -9input neuron is taught to output when the input (;,;< and ;-

is or : and to output : when the input is ::: or ::. Then, before applying

the firing rule, the truth table is7

1: 1 1 1 1

2: 1 1 1 1

3: 1 1 1 1

5&: 61 61 61 1 61 1

As an example of the way the firing rule is applied, take the pattern ::. !t differs

from ::: in element, from :: in < elements, from : in - elements and from

in < elements. Therefore, the 6nearest6 pattern is ::: which belongs in the :9

taught set. Thus the firing rule re0uires that the neuron should not fire when the

input is ::. 3n the other hand, : is e0ually distant from two taught patterns that

have different outputs and thus the output stays undefined (:=.

/y applying the firing in every column the following truth table is obtained7

1: 1 1 1 1

2: 1 1 1 1

3: 1 1 1 1

5&: 61 61 1 1 1

The difference between the two truth tables is called thegeneralisation of the

neuron.Therefore the firing rule gives the neuron a sense of similarity and enables

it to respond 6sensibly6 to patterns not seen during training.

1


11/32

,., Pattern 1e+o!nition - an eamle

An important application of neural networks is pattern recognition. 'atternrecognition can be implemented by using a feed9forward (figure neural

network that has been trained accordingly. &uring training, the network is

trained to associate outputs with input patterns. hen the network is used, it

identifies the input pattern and tries to output the associated output pattern.

The power of neural networks comes to life when a pattern that has no output

associated with it, is given as an input. !n this case, the network gives the

output that corresponds to a taught input pattern that is least different from

the given pattern.

Figure .

For example4

The network of figure is trained to recogni2e the patterns T and $. The

associated patterns are all black and all white respectively as shown next

11


12/32

!f we represent black s0uares with : and white s0uares with then the truth tables

for the - neurons after generali2ation are7

;4 : : : :

;


13/32

From the tables it can be seen the following associations can be extracted4

!n this case, it is obvious that the output should be all blacks since the input pattern

is almost the same as the 6T6 pattern.

$ere also, it is obvious that the output should be all whites since the input pattern

is almost the same as the 6$6 pattern.

13


14/32

$ere, the top row is < errors away from the T and - from an $. >o the top output is

black. The middle row is error away from both T and $ so the output is random.

The bottom row is error away from T and < away from $. Therefore the output is

black. The total output of the network is still in favour of the T shape.

,.4 A more +omli+ated neuron

The previous neuron doesn6t do anything that conventional computers don6t do

already. A more complicated neuron (figure


15/32

Architecture o% neural networks

4." eed-for'ard net'ors

Feed9forward ANNs (figure 9 allow signals to travel one way only7 from input

to output. There is no feedback (loops i.e. the output of any layer does not affect

that same layer. Feed9forward ANNs tend to be straight forward networks that

associate inputs with outputs. They are widely used in pattern recognition. This

type of organi2ation is also referred to as bottom9up or top9down.

4.2 eedba+ net'ors

Feedback networks (figure 9


16/32

Figure . An example of a simplefeedforward network

Figure .< An example of a complicated

network

4., &et'or layers

The commonest type of artificial neural network consists of three groups, or layers,

of units4 a layer of 1in#ut1 units is connected to a layer of 1hidden1 units, which is

connected to a layer of 7out#ut1 units. (Figure .

The activity of the input units represents the raw information that is fed into the

network.

The activity of each hidden unit is determined by the activities of the input units

and the weights on the connections between the input and the hidden units.

The behavior of the output units depends on the activity of the hidden units and

the weights between the hidden and output units.

1/


17/32

This simple type of network is interesting because the hidden units are free to

construct their own representations of the input. The weights between the input and

hidden units determine when each hidden unit is active, and so by modifying these

weights, a hidden unit can choose what it represents.

e also distinguish single9layer and multi9layer architectures. The single9layer

organi2ation, in which all units are connected to one another, to form the most

general case and is of more potential computational power than hierarchically

structured multi9layer organi2ations. !n multi9layer networks, units are often

numbered by layer, instead of following a global numbering.

10


18/32

4.4 Per+etions

The most effective thing on the work on neural nets in the *:6s went under the

heading of 6perceptrons6 a term coined by Frank Cosenblatt. The perceptron (figure

. turns out to be an %+' model ( neuron with weighted inputs with some

additional, fixed, pre99processing. 5nits labelled A, A


19/32

. /he 'earning *rocess

The memori2ation of patterns and the subse0uent response of the network can be

categori2ed into two general paradigms4

Associati!e "a##ingin which the network learns to produce a particular pattern

on the set of input units whenever another particular pattern is applied on the set of

input units. The associative mapping can generally be broken down into two

mechanisms4

Auto-association4 an input pattern is associated with itself and the states of

input and output units coincide. This is used to provide pattern completition, ie to

produce a pattern whenever a portion of it or a distorted pattern is presented. !n thesecond case, the network actually stores pairs of patterns building an association

between two sets of patterns.

Hetero-association4 is related to two recall mechanisms4

Nearest-neighborrecall, where the output pattern produced corresponds to the

input pattern stored, which is closest to the pattern presented, and

Interpolativerecall, where the output pattern is a similarity dependent

interpolation of the patterns stored corresponding to the pattern presented. Eet

another paradigm, which is a variant associative mapping is classification, ie when

there is a fixed set of categories into which the input patterns are to be classified.

5egularity detectionin which units learn to respond to particular properties of

the input patterns. hereas in associative mapping the network stores the

relationships among patterns, in regularity detection the response of each unit has aparticular 6meaning6. This type of learning mechanism is essential for feature

discovery and knowledge representation.

8very neural network possesses knowledge which is contained in the values of the

connections weights. %odifying the knowledge stored in the network as a function

of experience implies a learning rule for changing the values of the weights.

1


20/32

!nformation is stored in the weight matrix of a neural network. #earning is the

determination of the weights. Following the way learning is performed, we can

distinguish two ma"or categories of neural networks4

(i+ed networksin which the weights cannot be changed, ie d=dt:. !n such

networks, the weights are fixed a priori according to the problem to solve.

Ada#ti!e networkswhich are able to change their weights, ie d=dt not :.

All learning methods used for adaptive neural networks can be classified into two

ma"or categories4

u#er!ised learningwhich incorporates an external teacher, so that each output

unit is told what its desired response to input signals ought to be. &uring the

learning process global information may be re0uired. 'aradigms of supervised

learning include error9correction learning, reinforcement learning and stochastic

learning.An important issue conserving supervised learning is the problem of error

convergence, ie the minimi2ation of error between the desired and computed unit

values. The aim is to determine a set of weights which minimi2es the error. 3ne

well9known method, which is common to many learning paradigms, is the least

mean s0uare (#%> convergence.

2


21/32

8nsu#er!ised learninguses no external teacher and is based upon only local

information. !t is also referred to as self9organi2ation, in the sense that it self9

organi2es data presented to the network and detects their emergent collective

properties. 'aradigms of unsupervised learning are $ebbian learning and

competitive learning.

Anoigmoid

For linear units, the output activity is proportional to the total weighted output.

For threshold units, the output are set at one of two levels, depending on whether

the total input is greater than or less than some threshold value.

For sig"oid units, the output varies continuously but not linearly as the input

changes. >igmoid units bear a greater resemblance to real neurones than do linear

or threshold units, but all three must be considered rough approximations.

To make a neural network that performs some specific task, we must choose how

the units are connected to one another (see figure ., and we must set the weightson the connections appropriately. The connections determine whether it is possible

for one unit to influence another. The weights specify the strength of the influence.

21


22/32

e can teach a three9layer network to perform a particular task by using the

following procedure4

1. 7e present the network with training exa#ples' whichconsist of a pattern of activities for the inp$t $nits togetherwith the desired pattern of activities for the o$tp$t $nits.

2. 7e deter#ine how closely the act$al o$tp$t of the network#atches the desired o$tp$t.

3. 7e change the weight of each connection so that thenetwork prod$ces a better approxi#ation of the desiredo$tp$t.

5.2 An Example to illustrate the above teaching

procedure:

Assume that we want a network to recogni2e hand9written digits. e might use an

array of, say,


23/32

Another way to calculate the 8 is to use the /ack9propagation algorithm which

is described below, and has become nowadays one of the most important tools for

training neural networks.

5.3 The ac!"#ropagation Algorithm

!n order to train a neural network to perform some task, we must ad"ust the weights

of each unit in such a way that the error between the desired output and the actual

output is reduced. This process re0uires that the neural network compute the error

derivative of the weights ()W. !n other words, it must calculate how the error

changes as each weight is increased or decreased slightly. The back propagation

algorithm is the most widely used method for determining the )W.

The back9propagation algorithm is easiest to understand if all the units in the

network are linear. The algorithm computes each )Wby first computing the )A,the rate at which the error changes as the activity level of a unit is changed. For

output units, the )Ais simply the difference between the actual and the desired

output. To compute the)Afor a hidden unit in the layer "ust before the output

layer, we first identify all the weights between that hidden unit and the output units

to which it is connected. e then multiply those weights by the )As of those

output units and add the products. This sum e0uals the )Afor the chosen hidden

unit. After calculating all the )As in the hidden layer "ust before the output layer,

we can compute in like fashion the )As for other layers, moving from layer to

layer in a direction opposite to the way activities propagate through the network.

This is what gives back propagation its name. 3nce the )Ahas been computed fora unit, it is straight forward to compute the)Wfor each incoming connection of

the unit. The)Wis the product of the 8A and the activity through the incoming

connection.

Note that for non9linear units, the back9propagation algorithm includes an extra

step. /efore back9propagating, the )Amust be converted into the )I, the rate at

which the error changes as the total input received by a unit is changed.

23


24/32

./he back$#ro#agation Algorith" $ a "athe"atical a##roach ,non$linear-

5nits are connected to one another. +onnections correspond to the edges of the

underlying directed graph. There is a real number associated with each connection,

which is called the weight of the connection. e denote by i" the weight of the

connection from unit ui to unit u". !t is then convenient to represent the pattern of

connectivity in the network by a weight matrix whose elements are the weights

i". Two types of connection are usually distinguished4 excitatory and inhibitory.

A positive weight represents an excitatory connection whereas a negative weight

represents an inhibitory connection. The pattern of connectivity characteri2es the

architecture of the network.

A unit in the output layer determines its activity by following a two step procedure.

First, it computes the total weighted input x", using the formula4

24


25/32

where yi is the activity level of the "th unit in the previous layer and i" is the

weight of the connection between the ith and the "th unit.

Next, the unit calculates the activity y" using some function of the total weighted

input. Typically we use the sigmoid function4

3nce the activities of all output units have been determined, the network computes

the error 8, which is defined by the expression4

here y" is the activity level of the "th unit in the top layer and d" is the desired

output of the "th unit.

The back9propagation algorithm consists of four steps4

. +ompute how fast the error changes as the activity of an output unit is changed.

This error derivative (8A is the difference between the actual and the desired

activity.


26/32

. +ompute how fast the error changes as the activity of a unit in the previous layer

is changed. This crucial step allows back propagation to be applied to multilayernetworks. hen the activity of a unit in the previous layer changes, it affects the

activates of all the output units to which it is connected. >o to compute the overall

effect on the error, we add together all these separate effects on output units. /ut

each effect is simple to calculate. !t is the answer in step < multiplied by the weight

on the connection to that output unit.

/y using steps < and , we can convert the 8As of one layer of units into 8As forthe previous layer. This procedure can be repeated to get the 8As for as many

previous layers as desired. 3nce we know the 8A of a unit, we can use steps < and

- to compute the 8s on its incoming connections.

2/


27/32

#. Ali+ations of neural net'ors

#." &eural &et'ors in Pra+ti+e

Hiven this description of neural networks and how they work, what real world

applications are they suited forI Neural networks have broad applicability to real

world business problems. !n fact, they have already been successfully applied in

many industries.

>ince neural networks are best at identifying patterns or trends in data, they are

well suited for prediction or forecasting needs including4

eather forecasting

!ndustrial process control

+ustomer research

&ata validation

Cisk management

Target marketing

/ut to give you some more specific examples7 ANN are also used in the following

specific paradigms4 recognition of speakers in communications7 diagnosis of

hepatitis7 recovery of telecommunications from faulty software7 interpretation of

multimeaning +hinese words7 undersea mine detection7 texture analysis7 three9

dimensional ob"ect recognition7 hand9written word recognition7 and facialrecognition. .

20


28/32

#.2 &eural net'ors in medi+ine

Artificial Neural Networks (ANN are currently a 6hot6 research area in medicine

and it is believed that they will receive extensive application to biomedical systems

in the next few years. At the moment, the research is mostly on modeling parts of

the human body and recogni2ing diseases from various scans (e.g. cardiograms,

+AT scans, ultrasonic scans, etc..

Neural networks are ideal in recogni2ing diseases using scans since there is no

need to provide a specific algorithm on how to identify the disease. Neural

networks learn by example so the details of how to recogni2e the disease are not

needed. hat is needed is a set of examples that are representative of all the

variations of the disease. The 0uantity of examples is not as important as the

60uantity6. The examples need to be selected very carefully if the system is to

perform reliably and efficiently.

$.2.1 %odeling and &iagnosing the 'ardiovascular ()stem

Neural Networks are used experimentally to model the human cardiovascularsystem. &iagnosis can be achieved by building a model of the cardiovascular

system of an individual and comparing it with the real time physiological

measurements taken from the patient. !f this routine is carried out regularly,

potential harmful medical conditions can be detected at an early stage and thus

make the process of combating the disease much easier.

A model of an individual6s cardiovascular system must mimic the relationship

among physiological variables (i.e., heart rate, systolic and diastolic blood

pressures, and breathing rate at different physical activity levels. !f a model is

adapted to an individual, then it becomes a model of the physical condition of that

individual. The simulator will have to be able to adapt to the features of any

individual without the supervision of an expert. This calls for a neural network.

Another reason that "ustifies the use of ANN technology is the ability of ANNs to

provide sensor fusion which is the combining of values from several different

sensors. >ensor fusion enables the ANNs to learn complex relationships among the2


29/32

individual sensor values, which would otherwise be lost if the values were

individually analy2ed. !n medical modeling and diagnosis, this implies that even

though each sensor in a set may be sensitive only to a specific physiological

variable, ANNs are capable of detecting complex medical conditions by fusing the

data from the individual biomedical sensors.

$.2.2 Electronic noses

ANNs are used experimentally to implement electronic noses. 8lectronic noses

have several potential applications in telemedicine. Telemedicine is the practice of

medicine over long distances via a communication link. The electronic nose would

identify odours in the remote surgical environment. These identified odours would

then be electronically transmitted to another site where a door generation system

would recreate them. /ecause the sense of smell can be an important sense to the

surgeon, telesmell would enhance telepresent surgery.

$.2.3 *nstant #h)sician

An application developed in the mid9)D:s called the 1instant physician1 trained

an autoassociative memory neural network to store a large number of medical

records, each of which includes information on symptoms, diagnosis, and

treatment for a particular case. After training, the net can be presented with input

consisting of a set of symptoms7 it will then find the full stored pattern that

represents the 1best1 diagnosis and treatment.

#., &eural &et'ors in business

/usiness is a diverted field with several general areas of speciali2ation such as

accounting or financial analysis. Almost any neural network application would fit

into one business area or financial analysis.

There is some potential for using neural networks for business purposes, including

resource allocation and scheduling. There is also a strong potential for using neural

networks for database mining that is, searching for patterns implicit within the

explicitly stored information in databases. %ost of the funded work in this area is

classified as proprietary. Thus, it is not possible to report on the full extent of the

work going on. %ost work is applying neural networks, such as the $opfield9Tank

network for optimi2ation and scheduling.

2


30/32

$.3.1 %ar!eting

There is a marketing application which has been integrated with a neural network

system. The Airline %arketing Tactician (a trademark abbreviated as A%T is a

computer system made of various intelligent technologies including expertsystems. A feed forward neural network is integrated with the A%T and was

trained using back9propagation to assist the marketing control of airline seat

allocations. The adaptive neural approach was amenable to rule expression.

Additionally, the application6s environment changed rapidly and constantly, which

re0uired a continuously adaptive solution. The system is used to monitor and

recommend booking advice for each departure. >uch information has a direct

impact on the profitability of an airline and can provide a technological advantage

for users of the system. J$utchison K >tephens, )DLM

hile it is significant that neural networks have been applied to this problem, it isalso important to see that this intelligent technology can be integrated with expert

systems and other approaches to make a functional system. Neural networks were

used to discover the influence of undefined interactions by the various variables.

hile these interactions were not defined, they were used by the neural system to

develop useful conclusions. !t is also noteworthy to see that neural networks can

influence the bottom line.

$.3.2 'redit Evaluation

The $N+ +ompany, founded by Cobert $echt9Nielsen, has developed several

neural network applications. 3ne of them is the +redit >coring system which

increases the profitability of the existing model up to


31/32

4. Conclusion

The computing world has a lot to gain from neural networks. Their ability to learn

by example makes them very flexible and powerful. Furthermore there is no need

to devise an algorithm in order to perform a specific task7 i.e. there is no need tounderstand the internal mechanisms of that task. They are also very well suited for

real time systems because of their fast response and computational times which are

due to their parallel architecture.

Neural networks also contribute to other areas of research such as neurology and

psychology. They are regularly used to model parts of living organisms and to

investigate the internal mechanisms of the brain.

'erhaps the most exciting aspect of neural networks is the possibility that some day

6conscious6 networks might be produced. There is a number of scientists arguingthat consciousness is a 6mechanical6 property and that 6conscious6 neural networks

are a realistic possibility.

Finally, ! would like to state that even though neural networks have a huge

potential we will only get the best of them when they are integrated with

computing, A!, fu22y logic and related sub"ects.

31


32/32

5e%erences:

1. An introd$ction to ne$ral co#p$ting. Aleksander' ). and8orton' ,. 2nd edition2. "e$ral "etworks at +acific "orthwest "ational 9aboratory

http:66www.e#sl.pnl.gov:26docs6cie6ne$ral6ne$ral.ho#epage.ht#l

3. )nd$strial Applications of "e$ral "etworks research reports;sprit' ).(.*roall'

3836801 Neural Networks

Documents

Transcript of 3836801 Neural Networks