Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific...
Transcript of Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific...
![Page 1: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/1.jpg)
Internet EngineeringJacek Mazurkiewicz, PhD
Softcomputing
Part 1: Introduction, Elementary ANNs
![Page 2: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/2.jpg)
Formal Introduction• contact hours, room No. 225 building C-3:
Monday: 13:00 – 15:00Thursday: 11:00 – 13:00
• slides: www.zsk.ict.pwr.wroc.pl
• „Professor Wiktor Zin”
• test: 27.01.2020 during lecture time
- softcomputing:
- lecture + project
- project mark – 20% of final mark, bonus question!
![Page 3: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/3.jpg)
Softcomputing Lecture Schedule• Lecture 01 – 07th October 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 02 – 21st October 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 03 – 28th October 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 04 – 04th November 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 05 – 05th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3
• Lecture 06 – 12th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3
• Lecture 07 – 13th November 2019 – Wednesday, 11:15 - 13:00, room 21 C-3
• Lecture 08 – 18th November 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 09 – 19th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3
• Lecture 10 – 25th November 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 11 – 26th November 2019 – Tuesday, 11:15 - 13:00, room 21 C-3
• Lecture 12 – 02nd December 2019 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 13 – 03rd December 2019 – Tuesday, 11:15 - 13:00, room 21 C-3
• Lecture 14 – 13th January 2020 – Monday, 11:15 - 13:00, room 21 C-3
• Lecture 15 – 27th January 2020 – Monday, 11:15 - 13:00, room 21 C-3
![Page 4: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/4.jpg)
Program• Idea of intelligent processing
• Fuzzy sets and approximate reasoning
• Expert systems - knowledge base organization
• Expert systems - reasoning rules creation
• Expert systems: typical organization and applications
• Artificial neural networks: learning and retrieving algorithms
• Multilayer percetpron, RBF
• Kohonen neural network, CNN
• Hopfield neural network
• Hamming neural network
• Artificial neural networks: applications
• Genetic algorithms: description and classification
• Genetic algorithms: basic mechanisms and solutions
![Page 5: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/5.jpg)
SUBJECT OBJECTIVES
C1. Knowledge of artificial neural networks in pattern recognition, digital signals
and data processing: topology of networks, influence of parameters for network behavior.
C2. Knowledge of genetic algorithms used for data pre- and postprocessing.
C3. Knowledge of expert systems – reasoning rules and knowledge base creation for different tasks.
C4. Skills of special environment usage for project phase, modeling and simulation
of softcomputing systems in case of different scientific problems.
SUBJECT EDUCATIONAL EFFECTS
relating to knowledge:
PEK_W01 – knows the rules and the idea of intelligent processing.
PEK_W02 – defines the fuzzy sets and understands the idea of approximate reasoning.
PEK_W03 – defines the knowledge base and reasoning rules, knows the expert systems construction.
PEK_W04 – knows the architecture of typical artificial neural networks structures, learning and retrieving algorithms,
applications.
PEK_W05 – knows the description, classification, examples of applications of genetic algorithms
relating to skills:
PEK_U01 – can use the environments for project phase, modeling and simulation of artificial neural networks
as well as genetic algorithms in different tasks about pattern digital signals recognition.
PEK_U02 – can use the environments for project phase, modeling and implementation of expert systems
to dedicated fields of knowledge.
PEK_U03 – can use the environments for project phase, modeling and implementation of fuzzy sets and fuzzy reasoning
to dedicated fields of knowledge.
![Page 6: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/6.jpg)
Literature
• B. Bouchon Meunier, Fuzzy Logic and Soft Computing
• O. Castilo, A. Bonarini, Soft Computing Applications
• M. Caudill, Ch. Butler, Understanding Neural Networks
• E. Damiani, Soft Computing in Software Engineering
• R. Hecht-Nielsen, Neurocomputing
• S. Y. Kung, Digital Neural Networks
• D. K. Pratihar, Soft Computing
• S. N. Sivanandam, S. N. Deepa, Principles of Soft Computing
• A. K. Srivastava, Soft Computing
• D. A. Waterman, A Guide to Expert Systems
• D. Zhang, Parallel VLSI Neural System Design
![Page 7: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/7.jpg)
Why Neural Networks and Company?
Still in active use
No chance to solve some problems in other way
Human ability vs. classical programs
Works as primitive human’s brain
Artificial intelligence has power!
ANN + Fuzzy Logic + Expert Systems + Rough Sets + Ant Algorithms
= SoftComputing
![Page 8: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/8.jpg)
The Story1943 – McCulloch & Pitts
– model of artificial neuron
1949 – Hebb– information stored by biological neural nets
1958 – Rosenblatt– perceptron model
1960 – Widrow & Hoff– first neurocomputer - Madaline
1969 – Minsky & Papert– XOR problem – single-layer perceptron limitations
1986 – McCleland & Rumelhart– backpropagation algorithm
![Page 9: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/9.jpg)
Where Softcomputing is in Use?
Letters, signs, characters, digits recognition
Recognition of ship types – data from sonar
Electric power prediction
Different kinds of simulators and computer games
Engine diagnostic – in planes, vehicles
Rock-type identification
Bomb searching devices
![Page 10: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/10.jpg)
Neural Networks Realisation
Set of connected identical neurons
Artificial neuron based on a biological neuron
Hardware realisation – digital device
Software realisation – simulators
Artificial neural network – idea, algorithm, mathematical formulas
Works in parallel
No programming – learning process necessary
![Page 11: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/11.jpg)
Learning
With a Teacher
Without a Teacher
Klasyfikator
Wektor cech
(dane nauki)
Wynik
klasyfikacji
NauczycielTeacher
Learningvector
Parameters Weights
Result oflearning
Klasyfikator
Wektor cech
(dane testowe)
Wynik
klasyfikacjiLearning
vectorResult oflearning
Parameters Weights
![Page 12: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/12.jpg)
Softcomputing vs. Classical Computer
Different limitations of softcomputing methods
No softcomputing:
– operations based on symbols: editors, algebraic equations
– calculations with a high level of precision
Softcomputing is very nice, but not as universal as computer
![Page 13: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/13.jpg)
Anatomy Foundations (1)Nervous System – 2-ways, symmetrical
set of structures, divided into 4 parts:
Spinal Cord– receiving and transmission of data
Prolonged Cord– breathing, blood system, digestion
Cerebellum– movement control
Brain (ca. 1.3 kg) – 2 hemispheres– feeling, thinking, movement
brain
brain sterncerebellum
prolonged cord
spinal cord
nervous system
![Page 14: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/14.jpg)
Anatomy Foundations (2)
![Page 15: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/15.jpg)
Anatomy Foundations (3)
Cerebral cortex – thickness: 2 mm, area: ca. 1.5 m2
Cerebral cortex divided into 4 part – lobes
Each lobe is corrugated
Each hemisphere is responsible for half part of body:right for left part, left for right part
Hemispheres are identical in case of a structure, buttheir functions are different
![Page 16: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/16.jpg)
Anatomy Foundations (4)
Brain composed by fibres with large number of branches
Two types of cells in nervous tissue: neurons and gley cells
There are more gley cells:– no data transfer among neurons– catering functions
Ca. 20 milliard neurons in cerebral cortex
Ca. 100 milliard neurons in whole brain
Neuron: dendrites – inputs, axon – output, body of neuron
Neuron: thousands of synapses – connections to other neurons
![Page 17: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/17.jpg)
Anatomy Foundations (5)
Neurons in work:
• chemical-electrical signal transferring
• cell generates electrical signals
• electric pulse is changed into a chemical signal at the end of axon
• chemical info passed by neurotransmitters
• 50 different types of neurons
• neurons driven by a frequency of hundreds of Hz
• neurons are rather low devices!
![Page 18: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/18.jpg)
Anatomy Foundations (6)
![Page 19: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/19.jpg)
Biological and Artificial Neural Nets
Artificial neural networks are a good solution for:– testing already identified biological systems– pattern recognition– alternative configurations to find the basic features of them
Artificial neural networks are primitive brothers of biological nets
Biological nets have sophisticated internal features important for their normal work
Biological nets have sophisticated time dependences ignored in most artificial networks
Biological connections among neurons are different and complicated
Most architectures of artificial nets are unrealistic from the biology point of view
Most learning rules for artificial networks are unreal in biology point of view
Most biological nets we can compare to already learned artificial nets to realise function described in a very detailed way
![Page 20: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/20.jpg)
Linear ANN - ADALINE (ADAive Linear Neuron)single neuron’s answer:
+...
x
x
x
1
2
M
w
w
w
1
2
M
w0
y
1
=
+=M
j
jj wxwy1
0
M – number of input neurons
K – number of output neurons
=
==M
j
jj xwy0
~)~( xwxT
),...,,(~10 Mxxxcol=x
10 =x
),...,,( 10 Mwwwcol=w
scalar description vector description
multi-output net:
![Page 21: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/21.jpg)
Single-Layer Multi-Output Network
1 x x x1 2 M
y y y1 2 K
w w w10 20 K0
w w w11 12 1K
w w w21 22 2Kw w wM1 M2 MK
Wkj
Outputneuron
Inputneuron
k-neuron’s answer:
=
=M
j
jkjK xwy0
)(x
column= xwy(x)T WXy(X) =
=
KMKK
M
M
www
www
www
10
22120
11110
W
![Page 22: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/22.jpg)
Learning Procedureexperimental data: N - series
Nxxx ,...,, 21
N
KKK ttt ,...,, 21
– learning data
– required answers
,N
K
N tx → – function implemented by net
error function – mean-square error:
( )( )= =
−=N
n
K
k
n
kk tyWE1 1
2
2
1)( w
= = =
−=
N
n
K
k
M
j
n
k
n
jjk txwWE1 1
2
02
1)(
looking for a minimum of E(W) function:
0)(
,
=
kjjk w
WE
![Page 23: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/23.jpg)
Pseudoinverse Algorithm
= =
=
−=
N
n
nj
M
j
nk
njkj
kj
xtxww
WE
1 0'' 02
2
1)( = = =
= N
n
M
j
N
n
nj
nk
nj
njkj xtxxw
jk 1 0' 1'
,
where:
=
N
M
N
M
M
xx
xx
xx
1
22
1
11
1
1
1
1
X
=
N
K
NN
K
K
ttt
ttt
ttt
21
22
2
2
1
11
2
1
1
T
=
KMKK
M
M
www
www
www
10
22120
11110
W
finally:
( ) TXWXXTTT = TXWT = TXX)(XW T1TT −=
rsepseudoinve−
= −
τ
T,XWτT
![Page 24: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/24.jpg)
Gradient-Type Algorithms (1)y
x
iterative approach:
xx
yxx
−=+1
steps:
– random weight vector
– new weight vector following: -w E
– repeat process generating the sequence of weights vectors:
– components of weight vectors calculated by
)(w
w
w
kj
kjkjw
Eww
−=
+ )()1(
error function: =n
nEE )()( ww = =
−=
K
k
M
j
n
k
n
jkj
n txwE1
2
02
1)(w
![Page 25: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/25.jpg)
Gradient-Type Algorithms (2)sequential approach:
kj
n
kjkjw
Eww
−=
+
)1( n
j
n
k
n
j
n
k
n
k
kj
n
xxtyw
E**)( =−=
x
error – delta rule: Widrow-Hoff rule:
n
k
n
k
n
k ty −= )(xn
j
n
kjkkj xww −=
+ )1(
algorithm: 1. set start values – by a random way for example
2. calculate a net answer for available xn
3. calculate an error value kn
4. calculate a new weight vector wkj(+1) according to the delta rule
5. repeat steps 2. – 4. until E less than required value
![Page 26: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/26.jpg)
Perceptron (1)x1
x2
xM
w0
wiM
wi2
wi1
yi
1
the story:
– Rosenblatt (1962)
– classification task
– Widrow & Hoff (1960) - ADALINE
answer: )()(0
xwTgxwgXy
M
j
jj =
=
=
w0 – threshold value
activation function:
−=
01
01)(
afor
aforag
=
00
01)(
afor
aforag
bipolar unipolar
=
=M
j
jj xwa0
![Page 27: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/27.jpg)
Perceptron (2)error function:
jkw
E
– does not exist, because g(a) is not differentiable
perceptron criterions:
compare the actual value of yi and the required output value di and:
– if yi = di the weights values of Wij and w0 are unchanged
– if yi = 0 and the required value di =1 update the weights as follow:
where: t – previous cycle, t+1 – actual cycle
– if yi = 1 and di = 0 update the weights according to:
where: bi – polarity, di – required neuron’s output signal
,)()1( jijij xtWtW +=+ ,1)()1( +=+ tbtb ii
,)()1( jijij xtWtW −=+ 1)()1( −=+ tbtb ii
![Page 28: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/28.jpg)
Perceptron (3)
=
−=p
k
k
i
k
i dyE1
2)()( ,)(
summarising:
1. look-up the input learning vectors
2. if classification is correct weights are not changed
3. if classification is wrong:
– if tn = +1 add xn to the weight values
– else subtract xn from the weight values
– value of is not important – can be set to 1, it only scales wi
![Page 29: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/29.jpg)
Perceptron – Problems (1)
linear separability: XOR problem – Minsky & Papert (1969):
y(x)=0C2
X1
X2
C1
In1 In2
Out
XOR
0 0 00 1 11 0 11 1 0
In In Out1 2
– non-linear separable problem
– solution: multilayer net
![Page 30: Jacek Mazurkiewicz, PhD Softcomputing · of softcomputing systems in case of different scientific problems. SUBJECT EDUCATIONAL EFFECTS relating to knowledge: PEK_W01 –knows the](https://reader034.fdocuments.us/reader034/viewer/2022042104/5e81cbe2dfc059758926ed24/html5/thumbnails/30.jpg)
Perceptron – Problems (2)
multilayer network for XOR problem solution:
w ww w
-2w w
= w =− w
=w
s1 s2
S