Artificial Neural Networksdataanalysis.vsb.cz/data/Vyuka/NAVY/01 ANN-History.pdfArtificial Neural...

NAVY Research Group Department of Computer Science

Faculty of Electrical Engineering and Computer Science VŠB-‐TUO 17. listopadu 15

708 33 Ostrava-‐Poruba Czech Republic

Artificial Neural Networks History and Basic Info

Ivan Zelinka

MBCS CIPT, www.bcs.org/ http://www.springer.com/series/10624

Department of Computer Science

Faculty of Electrical Engineering and Computer Science, VŠB-TUO 17. listopadu 15 , 708 33 Ostrava-Poruba

Czech Republic www.ivanzelinka.eu

navy.cs.vsb.cz 2

Topics

•  Our perception •  History of ANNs and basic terms •  Difference between ANNs and computer •  Biological and technical neuron •  Transfer functions •  ANNs classification •  ANN topology •  Linear and nonlinear separable problems •  Hilbert's 13th problem and the Kolmogorov theorem

navy.cs.vsb.cz 3

Objectives

The objectives of the lesson are: •  Explain effect of our perception by our brain •  Show historical background of ANNs •  Clarify the main difference between ANNs and computer •  Describe basic processes in biological and technical neuron •  Explain what are transfer functions and its impact on ANNs

classification and linear and nonlinear separable problems •  Define ANN topology anf its conjunctions with Hilbert's 13th problem

and the Kolmogorov theorem

navy.cs.vsb.cz 4

Artificial Neural Networks Our Perception

•  Is our perception reality or illusion? •  Neural networks are derived from the basic patterns - links observed

in the human brain. This provides both advantages and disadvantages of the human brain.

Memory-switch effect What do you see?

navy.cs.vsb.cz 5

Artificial Neural Networks Complexity

•  The human brain is very complicated neural system, which consists of biological neural networks, which are in terms of the primitive function at the lowest level.

•  Brain and complexity

navy.cs.vsb.cz 6

Artificial Neural Networks Stanislaw Lem the Futurologist

Stanisław Lem, 12 September 1921 – 27 March 2006) was a Polish writer of science fiction, philosophy and satire. His books have been translated into 41 languages and have sold over 45 million copies. From the 1950s to 2000s he published many books, both science fiction and philosophical/futurological. He is best known as the author of the 1961 novel Solaris, which has been made into a feature film three times.

navy.cs.vsb.cz 7

Artificial Neural Networks Does the Ocean Think?

•  Solaris (1961) by the Stanislaw Lem •  Non-human thinking system •  Planetary ocean as a brain

•  Fantasy and science fiction??? •  Really?

navy.cs.vsb.cz 8


•  Our brains has 100 billions interconnected cells

•  Microbiologist Yuri Gorby (http://faculty.rpi.edu/node/1179) –  Yuri is a pioneer in the emerging field of “electromicrobiology” and has

been heavily cited for his publications on electrically conductive protein filaments called ‘bacterial nanowires’. He established the first Electromicrobiology Group during his 5 years with the J. Craig Venter Laboratory in San Diego, CA, and was a founding member of the Electromicrobiology Group with his colleagues at the University of Southern California.

navy.cs.vsb.cz 9


•  Microbiologist Yuri Gorby –  Ocean bacteria has 100 trillions trillions interconnected cells –  Bacterial nanowires are conducting electricity –  Firing is 5x faster than in human brain –  Estimation of size and time is 140 000 000 km2 and more than 100000

years –  Living organism included into this network –  This activity is running too much long to be only random event

navy.cs.vsb.cz 10


Does the Ocean Really Think?

See all at

Through the Wormhole with Morgan Freeman Does the Ocean Think? Season 05 Episode 05

https://www.youtube.com/watch?v=qCxgTzG0OTQ

navy.cs.vsb.cz 11

Artificial Neural Networks Does the Swarm System Think?

•  Invincible by the Stanislaw Lem •  Non-human thinking system •  Swarm system as a brain •  Fantasy and science fiction??? •  Really?

navy.cs.vsb.cz 12

Artificial Neural Networks Invincible by the Stanislaw Lem

•  Swarm robotic •  Collective memory •  Decentralized control

•  See also –  http://news.bbc.co.uk/2/hi/science/nature/8044200.stm –  http://imr.ciirc.cvut.cz/Swarm/Swarm –  http://mrs.felk.cvut.cz/research/swarm-robotics

navy.cs.vsb.cz 13

Artificial Neural Networks History

•  Ancient Egypt

navy.cs.vsb.cz 14

Artificial Neural Networks The Difference Between ANN and Computer

ANN Computer

It is learned by adjusNng of weights, thresholds and structures

It is programmed instrucNons (if, then, go to ...)

Memory and execuNve elements are arranged altogether

The process and memory for him are separated

Parallelism SequenNal

Tolerate deviaNons from the original informaNon

Does not tolerate deviaNons

Self-‐organizaNon during learning Unchanged program structure

navy.cs.vsb.cz 15

Artificial Neural Networks ANN Use

The use of neural networks is really wide and becomes more and more important. They can be used for example to •  Identification of data structures, radar or sonar signals,… •  Prediction of behavior •  Classification •  Optimization •  Filtration

navy.cs.vsb.cz 16

Artificial Neural Networks ANN History

The history of a continual development of neural networks in the first half of the 20th century, when it was first published work of neurons and their American model. W. S. McCulloch. In the 40 years of this century, with his student W. Pitts neuron model developed, which is practically used today. •  1920s W. S. McCulloch •  1958 F. Rosenblatt •  1969 M. Minsky spolu & S. Papert - Perceptron •  1980s D. Rumelhart, G. Hinton a R. Wiliams „Learning Internal

Representation by Error Propagation” •  ANNs: Hopfield, Kohonen (SOM), Grossberg (ART) •  ANN and unconventional hybridization with bio-inspired algorithms

and fuzzy logic

navy.cs.vsb.cz 17

Artificial Neural Networks ANN – a Few Facts

Definition: The ability of neural networks to generalize results come from their own topology. Neural network is connection of a large number of simple units and it is this large number of their mutual connections that gives them this property. •  Human brain contain 1013-15 neurons •  10 000 neurons died per day (0.00025 % of total amount) •  Each neuron has an average of 6000 somatic and dendritic

connections covering about 40% of its surface.…

navy.cs.vsb.cz 18


Technicall neuron is basically a simple unit that evaluate-multiply all inputs to their weights (weights change during learning - hence the adaptation of the network), and the resulting values are added together. The resulting value is entered into the transfer function of the neuron (generally speaking, is a function which governs the response to input stimuli) and the output of this function is the output of the neuron, which serves as input to the next neuron.

That is the whole "miracle."

navy.cs.vsb.cz 19

Artificial Neural Networks ANN – Scheme of Biological Neuron

Biological neuron

navy.cs.vsb.cz 20

Artificial Neural Networks ANN – Scheme of Technical Neuron

Technical neuron

x1 w1

x2 w2

x3 w3

xn wn

...

y … output

1

n

i iix w

=∑

k wb … threshold

Inpu

t

navy.cs.vsb.cz 21


•  Biologically neuron is "a little more" complicated. In the description of the block also consists of inputs (dendrites), the body (soma) and outputs (axons). Its functions and is more complex. Each neuron is covered by the bilayer membrane lipids of the thickness of about 20 nm. In this membrane are embedded proteins that act as channel carrying ions. When channels open there is a change of membrane potential and the potential creation of a wave - the excitement is spreading through the body of the neuron dendrites to the axon.

•  Synapse acts as a valve and that in two ways - excitation (excitation) and inhibitory (damping). What will be the response (excitation, inhibition) depends on the transmitters that are responsible for our memory.

navy.cs.vsb.cz 22

Artificial Neural Networks ANN - Classification

Definition: All types of neural networks are composed of the same structural units - neurons. May contain different transfer functions, connected with each other and adaptive - learning algorithm. All this determines kind of network. ANNs are classified according to basic three criteria. •  According to the number of ANN layers:

–  One layer (Hopfield, Kohonen network, ...) –  Multiple layer (ART network, Perceptron, classical multilayer network

with backpropagation algorithm)

navy.cs.vsb.cz 23

Artificial Neural Networks ANN - Classification

•  Depending on the type of ANN learning algorithm: –  Supervised (network with backpropagation algorithm, ...) –  Unsupervised (Hopfield, ...)

•  According to the learning style of the ANN learning: –  Deterministic (e.g. Backpropagation algorithm) –  Stochastic (random setting their weights)…

navy.cs.vsb.cz 24

Artificial Neural Networks ANN - Transfer functions

Definition: Transfer function of the neuron is a function that transforms the input signal to the output signal between [0, 1] or [-1, +1]. This feature can be jump or combined and must be monotonic – i.e. that the associated responses of output to input is clear. The choice of function depends on the problem you want to solve. Among the most popular transfer function are: •  Perceptron •  Binary •  Logistic (It is also used the term "sigmoid") •  Hyperbolic tangent

navy.cs.vsb.cz 25

Artificial Neural Networks ANN - Perceptron transfer functions

It is a linear function, which was used in the first Rosenblatt neural network, and which because of its linearity was able to solve only linearly separable problems. This ultimately led to the decline of interest in neural networks.

( ) 0 x = 0F P x x else= ∀ >

navy.cs.vsb.cz 26

Artificial Neural Networks ANN - Binary transfer functions

As seen from the registration image and, this is a two valued function, which can have the value 0 or 1.

( ) 1 0 x = 0F B x else= ∀ >

navy.cs.vsb.cz 27

Artificial Neural Networks ANN - Logistic transfer functions

It is one of the most used functions, which was derived as an approximation of the transfer function of a biological neuron.

1( )1 xF Se−

=+

navy.cs.vsb.cz 28

Artificial Neural Networks ANN - Hyperbolic tangent transfer functions

Hyperbolic tangent function is equivalent to the logistic function, with the difference that can take values from -1 to +1, which means that it provides, inter alia, greater linear portion around the origin, which also has its importance.

( )x x

x xe eTanh xe e

−

−

−=+

navy.cs.vsb.cz 29

Artificial Neural Networks ANN – All transfer functions

( )x x

x xe eTanh xe e

−

−

−=+

1( )1 xF Se−

=+

( ) 1 0 x = 0F B x else= ∀ >

( ) 0x = 0

F P x xelse

= ∀ >

navy.cs.vsb.cz 30

Artificial Neural Networks ANN – Topology and parameters

Definition: The term topology (sometimes structure) networks understand how they are interconnected individual neurons, layers and the number of inputs and outputs of the network. In the network topology can include parameters such as the type of transfer functions in layers, learning parameter, momentum, etc.

Um�lá inteligence I Úvod.

-18-

U vícevrstvých sítí platí, =e první vrstva je v=dy v�tvící, co= znamená, =e neurony ve vstupní

vrstv� pouze distribuují vstupní hodnoty do

dalCí vrstvy. Vzhledem k tomu, =e se jedná

obecn� o vícebodový vstup do sít�, tak se

mluví o vstupních (výstupních) vektorech

informací. Jak je z Obr 1.5-2 vid�t, tak po�et

neuron� v jednotlivých vrstvách je variabilní a

zále=í na eCeném problému. Pozd�ji uká=eme,

=e lze zhruba odhadnout �i pípadn� vypo�ítat

po�et neuron� pro ka=dou vrstvu, nicmén� ani

pozd�ji uvedené vzorce na výpo�et vhodného

po�tu neuron� nejsou vCelék.

Mnohem vhodn�jCí zp�sob, jak ur�it po�et

neuron�, je pou=ít sí, která si sama tento

po�et m�ní podle vývoje globální chyby. Tato metoda je podrobn�ji objasn�na v kapitole 3.2.

Ka=dý vstup do neuronu má piazenu tzv. váhu Wxy, co= je bezrozm�rné �íslo, které UíkáV,

jaký význam má daný vstup pro písluCný neuron (ne pro sí �i problém).

U�ící schopnost neuronových sítí spo�ívá práv� v mo=nosti m�nit vCechny váhy v síti podle

vhodných algoritm� na rozdíl od sítí biologických, kde je schopnost se u�it zalo=ena na mo=nosti

tvorby nových spoj� mezi neurony. Fyzicky jsou tedy ob� schopnosti se u�it zalo=eny na

rozdílných principech, nicmén� z hlediska logiky ne. V pípad� vzniku nového spoje-vstupu u

biologického neuronu je to stejné, jako kdy= v technické síti je spoj mezi dv�ma neurony

ohodnocen vahou s hodnotou 0 a tudí= jako vstup pro neuron do kterého vstupuje, neexistuje. V

okam=iku kdy se váha zm�ní z 0 na libovolné �íslo, tak se daný spoj zviditelní - vznikne.

N11 N12 N13

N21 N22 N23

N31 N32 N33

W11 W33

Vstupní vrstva

Skrytá vrstva

Výstupní vrstva

Výstupní vektor

Vstupní vektor

Toto je klasická vícevrstvá sí (konkrétn� 3 vrstvy po 3neuronech). Po�et vrstev a neuron� v nich m�=e býtr�zný, nap. 4 vrstvy a v ka=dé jiný po�et neuron�.

Obr. 1.5-1 Vícevrstvá sí�


-19-

1.6 Jak sí� funguje

V�ta 1.6-1 O fázích sít� a tídách.

Jeden ze základních pedpoklad pro funkci sít� je její nau�ení - adaptace na daný

problém. Adapta�ní - u�ící proces se skládá ze dvou fází - adapta�ní a aktiva�ní, b�hem

kterých se nastavují váhy sít�.

Tídou rozumíme mnoFinu, která zahrnuje X jedinc se spole�nou vlastností.

Nov� vytvoenou, ale také jakoukoliv nenau�enou neuronovou sí lze povaBovat za jakéhosi

technického novorozence, který nic neumí. Neumí rozeznávat, klasifikovat,... Aby se mohla

pouBívat, musí být nau�ena stejn� jako kterýkoliv Bivý jedinec (samozejm�, Be zde se mnoBství

informací a délka u�ení nedá porovnávat). Z tohoto d�vodu byly vyvinuty algoritmy, pomocí

kterých se písluKná sí dokáBe nau�it na danou mnoBinu informací. Algoritmus se obvykle d�lí

na dv� fáze a to na fázi aktiva�ní (vybavovací) a adapta�ní (u�ící), které ke své �innosti

potebují trénovací mnoBinu. Trénovací mnoBina je skupina vektor� obsahujících informace o

daném problému pro u�ení. Pokud u�íme sí s u�itelem, pak jsou to dvojice vektor� vstup -

výstup. JestliBe u�íme sí bez u�itele, pak trénovací mnoBina obsahuje jen vstupní vektory. Pokud

pouBíváme jen fázi aktiva�ní, pak mluvíme o vybavování. Tuto fázi pouBíváme samostatn� jen

tehdy, kdyB je sí nau�ena. Cyklické stídání obou fází je vlastní u�ení.

W11 W32

Vstupní vrstva(v�tvící)

Skrytá vrstva

Výstupní vrstva

W12

a)

W11 W33


Skrytá vrstva

Výstupní vrstva

b)

Obr. 1.5-2 Rzné topologie sítí


-19-

1.6 Jak sí� funguje

V�ta 1.6-1 O fázích sít� a tídách.

Jeden ze základních pedpoklad pro funkci sít� je její nau�ení - adaptace na daný

problém. Adapta�ní - u�ící proces se skládá ze dvou fází - adapta�ní a aktiva�ní, b�hem

kterých se nastavují váhy sít�.

Tídou rozumíme mnoFinu, která zahrnuje X jedinc se spole�nou vlastností.

Nov� vytvoenou, ale také jakoukoliv nenau�enou neuronovou sí lze povaBovat za jakéhosi

technického novorozence, který nic neumí. Neumí rozeznávat, klasifikovat,... Aby se mohla

pouBívat, musí být nau�ena stejn� jako kterýkoliv Bivý jedinec (samozejm�, Be zde se mnoBství

informací a délka u�ení nedá porovnávat). Z tohoto d�vodu byly vyvinuty algoritmy, pomocí

kterých se písluKná sí dokáBe nau�it na danou mnoBinu informací. Algoritmus se obvykle d�lí

na dv� fáze a to na fázi aktiva�ní (vybavovací) a adapta�ní (u�ící), které ke své �innosti

potebují trénovací mnoBinu. Trénovací mnoBina je skupina vektor� obsahujících informace o

daném problému pro u�ení. Pokud u�íme sí s u�itelem, pak jsou to dvojice vektor� vstup -

výstup. JestliBe u�íme sí bez u�itele, pak trénovací mnoBina obsahuje jen vstupní vektory. Pokud

pouBíváme jen fázi aktiva�ní, pak mluvíme o vybavování. Tuto fázi pouBíváme samostatn� jen

tehdy, kdyB je sí nau�ena. Cyklické stídání obou fází je vlastní u�ení.

W11 W32


Skrytá vrstva

Výstupní vrstva

W12

a)

W11 W33


Skrytá vrstva

Výstupní vrstva

b)

Obr. 1.5-2 Rzné topologie sítí

navy.cs.vsb.cz 31


navy.cs.vsb.cz 32


Beside standard structures, one can get very natural one by means of Darwinian evolution in computers…

navy.cs.vsb.cz 33

Artificial Neural Networks ANN – The dynamics of the network learning and classification

Definition: One of the basic assumptions of the network is its learning - adaptation to the problem. Adaptation - Learning process consists of two phases - the adaptation and activation during which the adjusted weights of the network. By term class we understood set that includes X individuals with common properties. •  Activation phase is the process by which the presentation of vector

information for input to recalculate networks over all joints including their ranking weight up to the exit, where they appear on the network response vector in the format of the output vector. When learning this vector is compared with the original vector (required, output) and the difference between the two vectors (local variations - error) is stored in memory variables.

navy.cs.vsb.cz 34

Artificial Neural Networks ANN – The dynamics of the network learning and classification

•  Adaptation phase is the process by which local error is minimized network so that the recalculated weights of individual joints direction from output to input for making the most similarities output response to the original vector.

•  Cyclic repetition of both phases is called epoch. ANN are learned in epochs.

navy.cs.vsb.cz 35

Artificial Neural Networks ANN – Distance measurement

Definition: Distance measurement is used to determine to which class of the currently processed vector belongs. There is a few methods how to measure distances: •  Hamming •  Euclid •  Manhattan •  Square

navy.cs.vsb.cz 36

Artificial Neural Networks ANN – Hamming distance measurement

Calculation by this method is mainly used for binary vectors (Hopfield network). This distance is given by the sum of the elements of the two vectors.

[ ] [ ]1 2 1 2

1

... ...n n

n

i ii

A a a a B b b b

H a b=

= =

= −∑

navy.cs.vsb.cz 37

Artificial Neural Networks ANN – Euclid distance measurement

It is a distance calculation based on triangle. Formulas for calculating distances in 2D and ND (N-dimensional space) are.

2 21 2

2

1

2

( ( ) ( ))n

i

E X X for D

E A i B i for ND=

= Δ + Δ

= −∑

navy.cs.vsb.cz 38

Artificial Neural Networks ANN – Manhattan distance measurement

It's a simple modification of the Euclidean distance, which means a quick calculation at the cost of larger errors.

1

( ) ( )n

iM A i B i

=

= −∑

navy.cs.vsb.cz 39

Artificial Neural Networks ANN – Square distance measurement

This is again a simplification Manhattan distance, which in turn result in a higher rate, but even bigger mistake.

( ) ( )i

C MAX A i B i= −

navy.cs.vsb.cz 40

Artificial Neural Networks ANN – Training Set

Carrot

Cucumber

Apple

Pear

41 navy.cs.vsb.cz

Artificial Neural Networks ANN – Separable Problems


-20-

Aktiva�ní fáze je proces, pi kterém se pedlo>ený vektor informací na vstup sít� pepo�ítá pes

vCechny spoje v�etn� jejich ohodnocení vahami a> na výstup, kde se objeví odezva sít� na tento

vektor ve form� výstupního vektoru. Pi u�ení se tento vektor se porovná s vektorem originálním

(po>adovaným, výstupním) a rozdíl mezi ob�ma vektory (lokální odchylka - chyba) se ulo>í do

pam�ové prom�nné .

Adapta�ní fáze je proces, pi kterém je minimalizovaná lokální chyba sít� tak, >e se

pepo�ítávají váhy jednotlivých spoj� sm�rem z výstupu na vstup za ú�elem co nejv�tCí

podobnosti výstupní odezvy s originálním vektorem.

Po té se op�t opakuje aktiva�ní fáze. DalCí získaný rozdíl (lokální odchylka) se pi�te

k pedchozímu, atd... Pokud se tímto

postupem projde celá trénovací mno>ina,

je hotová jedna epocha. Celé sum�

odchylek za jednu epochu se íká

globální odchylka - chyba. Pokud je

globální odchylka menCí ne> námi

po>adovaná chyba, pak proces u�ení

skon�í.

Z výCe popsaného je vid�t, >e proces

u�ení není nic jiného, ne> pelévání

informací ze vstupu na výstup a naopak.

Pi u�ení se v tzv. vstupním prostoru

vytváejí shluky bod�, které pedstavují

jednotlivé �leny tíd, pi�em> ka>dý shluk

pedstavuje tídu (Obr. 1.6-1).

Pedstavte si, >e máte n�kolik tíd a to nap. tídu kol, aut, lodí, atd. Z ka>dé tídy vybereme

mno>inu reprezentativních zástupc� (vzor� pro u�ení) a vCechny popíCeme vhodným �íselným

zp�sobem ve form� vektor�. Pro ka>dou mno>inu vektor� jedné tídy vytvoíme vzorový vektor,

který bude zastupovat mateskou tídu z reálného sv�ta. V tomto okam>iku máme vytvoené

skupiny vektor� popisující jednotlivé �leny a jim písluCející pedstavitele tíd ve form� vektor�.

Napíklad máme vektory, které popisují vybrané �leny z tídy kol a vektor, který íká Sjá jsem

tída kolT. V tomto pípad� u�ení znamená to, >e se u�ící algoritmus sna>í najít takovou

Vstup 1

Vst

up 2

Tídy

Hranice tíd

Tída B

Tída C

Tída A

Obr. 1.6-1 Nelineární hranice mezi tídami - ideální pípad (?ádný z �len tíd A,B a C nele?í v prostoru druhé tídy) Z hlediska tvaru hranice je samozejm� ideální pímka.

0 2 4 6

0

2

4

6

Input 1

Border

Inpu

t 2

42 navy.cs.vsb.cz

Artificial Neural Networks ANN – Nonlinear Separable Problems


-24-

Tabulka 1.8-1 Data pro NLS

problém

1.9 T�ináctý Hilbert�v problém a Kolmogorov�v teorém

Jak u9 bylo nazna�eno, neuronová sí transformuje vstupní vektor na vektor výstupní.

Matematické koeny tohoto problému sahají a9 do roku 1900, kdy matematik D. Hilbert zveejnil

tzv. 13. Hilbert�v problém, který se týká nemo9nosti eNit obecn� rovnice 7. stupn� slo9ením

spojitých funkcí o dvou prom�nných, tzn. 9e koeny rovnice 1.9-1 se nedají vyjádit pomocí

jakékoliv kontinuální funkce jen o dvou parametrech [1].

Vstup 1

Vst

up 2

0

1

1

Výstup má hodnotu 0 Výstup má hodnotu 1

Problém je nelineárn� separabilní -hranicí nem�9e být pímka

Obr. 1.8-1 Problém nelineárn� separabilní

Vstup 1 Vstup 2 Výstup

0 0 0

1 0 1

0 1 1

1 1 0

Vstup 1

Vst

up 2

0

1

1


Problém je lineárn� separabilní - hranicí je pímka

Obr. 1.8-2 Problém lineárn� separabilní

Vstup1 Vstup 2 Výstup

0 0 1

1 0 0

0 1 1

1 1 1

Tabulka 1.8-2 - data pro LS problém

The border between two classes cannot be linear line

Input 1 Input 2 Output

0 0 0

1 0 1

0 1 1

1 1 0

43 navy.cs.vsb.cz

Artificial Neural Networks ANN – Linear Separable Problems

Input 1 Input 2 Output

0 0 1

1 0 0

0 1 1

1 1 1


-24-

Tabulka 1.8-1 Data pro NLS

problém

1.9 T�ináctý Hilbert�v problém a Kolmogorov�v teorém

Jak u9 bylo nazna�eno, neuronová sí transformuje vstupní vektor na vektor výstupní.

Matematické koeny tohoto problému sahají a9 do roku 1900, kdy matematik D. Hilbert zveejnil

tzv. 13. Hilbert�v problém, který se týká nemo9nosti eNit obecn� rovnice 7. stupn� slo9ením

spojitých funkcí o dvou prom�nných, tzn. 9e koeny rovnice 1.9-1 se nedají vyjádit pomocí

jakékoliv kontinuální funkce jen o dvou parametrech [1].

Vstup 1

Vst

up 2

0

1

1


Problém je nelineárn� separabilní -hranicí nem�9e být pímka

Obr. 1.8-1 Problém nelineárn� separabilní

Vstup 1 Vstup 2 Výstup

0 0 0

1 0 1

0 1 1

1 1 0

Vstup 1

Vst

up 2

0

1

1


Problém je lineárn� separabilní - hranicí je pímka

Obr. 1.8-2 Problém lineárn� separabilní

Vstup1 Vstup 2 Výstup

0 0 1

1 0 0

0 1 1

1 1 1

Tabulka 1.8-2 - data pro LS problém

The border between two classes is linear line.

Input 1

Inpu

t 2

navy.cs.vsb.cz 44

Artificial Neural Networks ANN – Hilbert's 13th problem and the Kolmogorov theorem

As already indicated, the neural network transforms the input vector to the vector output. Mathematical roots of the problem date back to 1900 when the mathematician D. Hilbert published the so-called Hilbert's 13th problem, which refers to the inability to solve the general equation of the seventh degree of the composition of continuous functions of two variables.

( )2 1

1 21 1

( , ,..., )n n

n j k jk j

f x x x h g xλ+

= =

⎛ ⎞= ⎜ ⎟

⎝ ⎠∑ ∑

navy.cs.vsb.cz 45

Artificial Neural Networks ANN – Hilbert's 13th problem and the Kolmogorov theorem

The neural network (three layers) that transforms input to output in the simple case of two inputs and one output contains the above equation in the first hidden layer (direction from output to input) 2n +1 neurons and in the next lower layer of n (2n +1 ) neurons. Other modifications it was found that by Kolmogorov theorem on problems of neural networks only results existential evidence that any solution to the problem of sufficient network of three layers.

navy.cs.vsb.cz 46

Artificial Neural Networks ANN – Questions

1.  Who and when created the first model of the neuron. 2.  When and by whom it was first constructed as a neural network was

called. 3.  Why has fallen interested in neural networks. 4.  What is the difference between the traditional PC and neural

networks. 5.  What passes generalization ability of neural networks. 6.  Explain the similarities between biological and technical neuron. 7.  How is divided into neural networks and what they mean different

labels. 8.  What types of transfer functions familiar to you, your relationships

and draw them.

navy.cs.vsb.cz 47

Artificial Neural Networks ANN – Questions

9.  Describe the stage of network learning and equipping know how and ongoing.

10. Why network terminates learning. 11. Explain in your own words what it is: era, the global error class. 12. What types of measuring distances in neural networks, you know,

describe and explain their significance. 13. Explain what does it means the term linear separable. 14. Explain in what follows Kolmogorov's theorem and its consequences.

navy.cs.vsb.cz 48

Conclusions

•  Principles and impact of our perception has been introduced •  History of ANNs and basic terms were explained •  Difference between ANNs and computer was described as well as

between biological and technical neuron •  Different kind of basic transfer functions were demonstrated •  Classification ANNs was explained •  ANN topology and its variability was discussed •  Linear and nonlinear separable problems were introduced in

connection with history and stagnation of ANNs •  Hilbert's 13th problem and the Kolmogorov theorem was mentioned

as well as its impact on ANNs history

49 navy.cs.vsb.cz

THANK YOU FOR YOUR ATTENTION

[email protected] www.ivanzelinka.eu

navy.cs.vsb.cz 50

Copyright

This didactic material is meant for the personal use of the student only, and is copyrighted. Its reproduction, even for a partial utilization, is strictly forbidden in compliance with and in force of the law on Authors rights.

Copyright©NAVY.CS.VSB.CZ

Artificial Neural Networksdataanalysis.vsb.cz/data/Vyuka/NAVY/01 ANN-History.pdfArtificial Neural...

Documents

Transcript of Artificial Neural Networksdataanalysis.vsb.cz/data/Vyuka/NAVY/01 ANN-History.pdfArtificial Neural...