![Unsupervised gene network inference with decision trees ... · Unsupervised gene network inference with decision trees and Random forests ... (e.g. [6–13]), usually achieving competitive](https://static.fdocuments.us/doc/165x107/5ec8fd42a1b3d77468653010/unsupervised-gene-network-inference-with-decision-trees-unsupervised-gene-network.jpg)
Competitive Networks, Clustering and unsupervised learning · 2019-09-01 · Competitive networks...
55
1 Department of Computer Engineering University of Kurdistan Neural Networks (Graduate level) Competitive Networks, Clustering and unsupervised learning By: Dr. Alireza Abdollahpouri
Transcript of Competitive Networks, Clustering and unsupervised learning · 2019-09-01 · Competitive networks...
1
Department of Computer Engineering
University of Kurdistan
Neural Networks (Graduate level) Competitive Networks,
Clustering and unsupervised learning
By: Dr. Alireza Abdollahpouri
Competitive networks
• Competition between neurons has been observed
in biological nerve systems
• To classify an input pattern into one of the m classes
– ideal case: one class node has output 1, all other 0 ;
– often more than one class nodes have non-zero output
2
Fixed-weight Competitive Nets
3
Maxnet
Consider the action of a MAXNET with four
neurons and inhibitory weights ε = 0.2.
4
/ / / /( ) ( ) ( ) ( )a a a a 1 2 3 40 02 0 04 0 06 0 0 8
/ / / /
/ / / /
/ / / /
/ / / /
/ / / /
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
a a a a
a a a a
a a a a
a a a a
a a a a
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 00 1 008 1 0 32 0 56
2 00 2 00 2 0192 0 48
3 00 3 00 3 0096 0 442
4 00 4 00 4 0008 0 442
5 00 5 00 5 00 0 421
If more than one node has a nonzero
activation, continue;
otherwise, stop.
Mexican hat
The Mexican Hat network [Kohonen, 1989a] is a more general
contrast enhancing subnet than the MAXNET. Three types of links can be found in such network:
- close neighbors: cooperative (mutually excitatory , w > 0)
- farther away neighbors: competitive (mutually inhibitory, w < 0)
- too far away neighbors: irrelevant (w = 0)
5
R1 R2
6
Mexican hat
Mexican hat- Algorithm
7
Step 0: Initialize parameters t_max, R1, R2 as desired Initialize weights:
Step 1: Present external signal s: x=s Save activation in array x_old (for i=1,…,n)
Set iteration counter: t = 1
Step 2: While t is less than t_max, do steps 3-7
8
Step 3: Compute net input (i = 1, …n)
Step 4: Apply activation function f (ramp from 0 to x_max, slope 1):
Mexican hat- Algorithm
9
Step 5: Save current activations in x_old:
Step 6: Increment iteration counter: t=t+1 Step 7: Test stopping condition: If t < t_max, continue; otherwise, stop.
Mexican hat- Algorithm
Mexican hat- Example
Using the Mexican Hat Algorithm We illustrate the Mexican
Hat algorithm for a simple net with seven units. The activation function for this net is:
10
11
Mexican hat- Example
12
Mexican hat- Example
13
Mexican hat- Example
14
Mexican hat- Example
15
Mexican hat- Example
16
Hamming Network
A Hamming net [Lippmann, 1987; DARPA, 1988] is a maximum likelihood classifier net that can be used to determine which of several exemplar vectors is most similar to an input vector (an n-tuple) Hamming distance of two vectors is the number of components in which the vector differ.
1 2
102
1 , 1 2) 3
0( #
1
HH x x H
n of dimentions
By setting the weights to be one-half the exemplar vector and setting the value of the bias to n/2, the net will find the unit with the closest exemplar simply by finding the unit with the largest net input.
17
Hamming Network
Hamming network-example
A Hamming net to cluster four vectors Given the exemplar vectors: e(1) = (1, -1, -1, -1, ), e(2) = (-1, -1, -1, 1)
18
Using the Hamming net to find the exemplar that is closest to each of the bipolar input pattern: (1, 1, -1, -1), (1, -1, -1, -1), (-1, -1, -1, 1), and (-1, -1, 1, 1)
19
Hamming network-example
20
Hamming network-example
21
Hamming network-example
22
Hamming network-example
23
Hamming network-example
Self Organized Map (SOM)
24
Proposed by: Teuvo Kohonen, Finland
Author of the book: “Self-organizing maps”, Springer (1997)
SOM assume a topological structure among the cluster units �This property is observed in the brain, but not found in other ANNs
"Almost every region of the body is represented by a corresponding region in both the primary motor cortex and the somatic sensory cortex" (Geschwind 1979:106).
25
Neurons dealing with closely related pieces of information are close together so that they can interact via short synaptic connections.
SOM: Inspiration
Retinotopy: in the visual system, adjacent spots on the retina are represented by adjacent neurons in the lateral geniculate nucleus and the primary visual cortex.
Tonotopy: in the auditory system, tones close to each other in terms of frequency are represented in topologically neighboring regions in the brain.
26
SOM: Inspiration
The bigger areas like your lips and hands represented here are the most sensitive.
"The finger tips of humans have the highest density of receptors: about 2500 per square cm!" (Kandel and Jessell 1991:374).
"small human" -schematization of body in primary motor and sensory cortices
27
SOM: Inspiration
2d array of neurons
Set of input signals (connected to all neurons in lattice)
Weighted synapses
x1 x2 x3 xn ...
wj1 wj2 wj3 wjn
j
28
SOM-Structure
Input vector
m clusters
Codewords
29
SOM-Structure
30
SOM- Concept
Initialization:
1- Weights are initialized with small random values.
2- Set topological neighborhood parameters
3-Set learning rate parameters
Repeat for a certain number of epochs t or until the map stops changing:
31
SOM -Algorithm
For each input instance: 1: Competition: Find the neuron whose weights are most similar to the input instance (winner). The neurons are competing to be the winner. 2: Cooperation: Determine the neighborhood of the winner neuron, in which all neurons should be excited for being reasonably similar to the input instance. These neurons can be seen as cooperating with each other. 3: Adaptation: Update the winner and its neighbors so that they are even more similar to the input instance.
In the learning algorithm, we have to find the neuron whose weights are most similar to the input instance (winner). How to determine the winner?
Similarity (euclidean distance):
32
SOM –Algorithm (Competition)
Neuron i with the minimum distance is the winner
2( ) ( )ij i
i
D j w x
According to neurobiological studies, similar neurons are close to each other in the cortex. When a neuron fires, its neighbors are often excited.
We also want our self-organizing map to have similar neurons close to each other
How to do that?
In SOM, not only the winning neuron will have its weights updated, but also its neighbors, although not as much as the winning neuron.
The size of the neighborhood should shrink over time.
33
SOM –Algorithm (Cooperation)
34
SOM- Neighborhood definition
linear array
rectangular matrix of cluster hexagonal matrix of cluster
time
35
Neighbourhood
36
Common Neighbourhood functions
It is maximal at the winning neuron, it is symmetrical about that neuron, it decreases monotonically to zero as the distance goes to infinity, and it is translation invariant (independent of the location of the winning neuron).
SOM- Adaptation
37
( ) ( ) [ ( )]ij ij i ijw new w old x w old
Weights should be updated in such a way that the new weights will be more similar (closer in distance) to the input.
Learning rate
• Input X larger than weight Wj , increase Wj • Input X smaller than weight Wj , decrease Wj
SOM-Example
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ /( ) , ( ) ( )t t 0 06 1 0 5
/ /
/ /
/ /
/ /
0 2 0 8
0 6 0 4
0 5 0 7
0 9 0 3
w R 0 /( ) 0 06
38
A Kohonen SOM to cluster 4 vectors:
The maximum number of clusters to be formed is m=2 (4 input nodes, 2 output nodes)
Suppose the learning rate is
Initialization
Random
No neighborhood
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
2 2 2 2
/ / / / /
2 2 2 2
/ / / / /
(1) (0 2 1) (0 6 1) (0 5 0) (0 9 0) 186
(2) (0 8 1) (0 4 1) (0 7 0) (0 3 0) 0 98
D
D
J 2
/ / /( ) ( ) [ ( )] ( )i i i i i iw new w old x w old w old x 2 2 2 206 04 06
/ /
/ /
/ /
/ /
0 2 0 92
0 6 0 76
0 5 0 28
0 9 0 12
w
/ /
/ /
/ /
/ /
0 2 0 8
0 6 0 4
0 5 0 7
0 9 0 3
w
39
SOM-Example
Distance calculation:
The i/p vector is closest to o/p node 2, so
The weights on the winning unit are updated
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ /
/ /
/ /
/ /
0 2 0 8
0 6 0 4
0 5 0 7
0 9 0 3
w
40
Distance calculation:
The i/p vector is closest to o/p node 1, so
The weights on the winning unit are updated
2 2 2 2
/ / / / /
2 2 2 2
/ / / / /
(1) (0 2 0) (0 6 0) (0 5 0) (0 9 1) 0 66
(2) (0 92 0) (0 76 0) (0 28 0) (0 12 1) 2 2768
D
D
1J
/ /
/ /
/ /
/ /
0 08 0 92
0 24 0 76
0 20 0 28
0 96 0 12
w
SOM-Example
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ /
/ /
/ /
/ /
0 2 0 8
0 6 0 4
0 5 0 7
0 9 0 3
w
41
Distance calculation:
The i/p vector is closest to o/p node 2, so
The weights on the winning unit are updated
2 2 2 2
/ / / / /
2 2 2 2
/ / / / /
(1) (0 08 1) (0 24 0) (0 2 0) (0 96 1) 18656
(2) (0 92 1) (0 76 0) (0 28 0) (0 12 0) 0 6768
D
D
J 2
/ /
/ /
/ /
/ /
0 08 0 968
0 24 0 304
0 20 0 112
0 96 0 048
w
SOM-Example
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ /
/ /
/ /
/ /
0 2 0 8
0 6 0 4
0 5 0 7
0 9 0 3
w
42
Distance calculation:
The i/p vector is closest to o/p node 1, so
The weights on the winning unit are updated
2 2 2 2
/ / / / /
2 2 2 2
/ / / / /
(1) (0 08 0) (0 24 0) (0 20 1) (0 96 1) 0 7056
(2) (0 968 0) (0 304 0) (0 112 1) (0 048 1) 2 724
D
D
1J
/ /
/ /
/ /
/ /
0 032 0 968
0 096 0 304
0 680 0 112
0 984 0 048
wEnd of first iteration
SOM-Example
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ / / 0 5 06 0 3
/ / /( ) ( ) [ ( )] ( )ij ij i ij ij iw new w old x w old w old x 0 3 0 7 0 3
Second Iteration
/ /
/ /
/ /
/ /
0 032 0 968
0 096 0 304
0 680 0 112
0 984 0 048
w
/ /
/ /
/ /
/ /
0016 0 980
0047 0 360
0 630 0055
0 999 0024
43
Reduce the learning rate
The weights update equations are now:
The weight matrix after the 2nd epoch of training is:
SOM-Example
( , , , ); ( , , , ); ( , , , ); ( , , , )1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
/ /
/ /
/ /
/ /
00 10
00 0 5
0 5 00
10 00
/ /
/ /
/ /
/ /
02 0 8
0 6 0 4
0 5 0 7
0 9 0 3
/ /
/ /
/ /
/ /
0 032 0 968
0 096 0 304
0 680 0 112
0 984 0 048
/ /
/ /
/ /
/ /
e
e
e
e
15 7 10000
4 6 7 0 3700
0 6300 5 4 7
10000 2 3 7
/ /
/ /
/ /
/ /
e
e
e
e
19 19 10000
5 7 15 0 4700
0 5300 6 6 15
10000 2 8 15
/ /
/ /
/ /
/ /
e
e
e
e
6 7 17 10000
2 0 16 0 4900
0 5100 2 3 16
10000 10 16
0 1 10 50 100
the average of the two vects in cluster 2
the average of the two vects in cluster 1
44
SOM-Example
The 100 input vectors are chosen randomly from within a circle of radius 0.5 (centered at the origin).
we assume a linear structure.
The initial learning rate is 0.5; it is reduced linearly to 0.01 over 100epochs
the winning unit and its nearest neighbor unit on either side (units J, J + 1, and J - 1) are allowed to learn.
45
SOM-Example
46
The initial weights are chosen randomly, with each component having a value between - 1 and 1.
There are 50 cluster units.
SOM-Example
47
cluster units after 10 epochs cluster units after 100 epochs
SOM-Example
Traveling Salesman Problem(TSP) The aim of the TSP is to find a tour of a given set
of cities that is of minimum length. A tour consists of visiting each city exactly once and returning to the starting city.
The net uses the city coordinates as input (n=2); there are as many cluster units as there are cities to be visited. The net has a linear topology (with the first and last unit also connected).
48
SOM-Example
49
Initial position of cluster units and location of cities
the result after 100 epochs of training with R=1 (learning rate decreasing from 0.5 to 0.4)
SOM-Example
ADEFGHIJBC
ADEFGHIJCB
50
Two candidate solutions
SOM-Example
is
has
likes to
Dove Hen Duck Goose Owl Hawk Eagle Fox Dog Wolf Cat Tiger Lion Horse Zebra Cow
Small 1 1 1 1 1 1 0 0 0 0 1 0 0 0 0 0 Medium 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0
Big 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1
2 legs 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 4 legs 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Hair 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
Hooves 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 Mane 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0
Feathers 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0
Hunt 0 0 0 0 1 1 1 1 0 1 1 1 1 0 0 0 Run 0 0 0 0 0 0 0 0 1 1 0 1 1 1 1 0 Fly 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0
Swim 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0
birds
peaceful
hunters
51
SOM Example -Clustering
Learning a one-dimensional representation of a two-dimensional (triangular) input space
0
25000
20 100
1000 10000 52
SOM-Example
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
W(1,j)
-1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
W(2
,j)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
W(1,j)
-1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
W(2
,j)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
W(1,j)
-1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
W(2
,j)
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
W(1,j)
-1-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
W(2
,j)
Initial random network After 100 steps
After 1000 steps After 10000 steps 53
SOM-Example
54
SOM-Example
Questions
