22.10.2005 1Prof. Pushpak Bhattacharyya, IIT Bombay

CS 621 Artificial Intelligence

Lecture 29 – 22/10/05

Prof. Pushpak Bhattacharyya

SOM, Theoretical Aspects Of Machine Learning, Probably

Approximately Correct Learning.

22.10.2005 2Prof. Pushpak Bhattacharyya, IIT Bombay

…

… … .

P neurons o/p layer

n neurons

Clusters:

AA :

A

Wp

B :

C :

: :

22.10.2005 3Prof. Pushpak Bhattacharyya, IIT Bombay

Wp(n+1) = Wp

(n) + η (I(n) – Wp(n))

Competitive learning: Weights to the winner neuron only are changed.

22.10.2005 4Prof. Pushpak Bhattacharyya, IIT Bombay

Generating function application to recurrence relation solving.

Example: Max. No of regions on 2-Dim plane produced by n lines.

22.10.2005 5Prof. Pushpak Bhattacharyya, IIT Bombay

L2L1L3

L4

0 x

y R1 = 2

R2 = R1 + 2

= 2+ 2

= 4

R3 = R2 + 3

= 4 + 3 = 7

R4 = 7 + 4 = 11 Rn = Rn-1 + n R1 = 2

22.10.2005 6Prof. Pushpak Bhattacharyya, IIT Bombay

Rn = Rn-1 + n

Rn-1 = Rn-2 + (n-1)

Rn-2 = Rn-3 + (n-2)

.

.

.

R2 = R1 + 2

22.10.2005 7Prof. Pushpak Bhattacharyya, IIT Bombay

Rn = n + (n-1) + (n-2)+ …..+R1 + 2

= n+(n-1)+(n-2)+…..+2+2

=n(n+1)/2 + 1

22.10.2005 8Prof. Pushpak Bhattacharyya, IIT Bombay

Generating Function Method

Key idea: Rns are generated by infinite

series function ,i.e,

F(x) = R1 + R2x + R3x2 + ………………+ Rnxn-1+ ….. ∞

Rn = Coeff of xn-1

22.10.2005 9Prof. Pushpak Bhattacharyya, IIT Bombay

Rn = Rn-1 + n

Rn - Rn-1 = n

R2 - R1 = 2

R3 – R2 = 3

.

.

(Continued)

22.10.2005 10Prof. Pushpak Bhattacharyya, IIT Bombay

F(x) = R1 + R2x + R3x2 + ………………+ Rnxn-1+

….. ∞ (1)

xF(x) = R1x + R2x2 + R3x3 + ……………+ Rnxn+

….. ∞ (2)

1 - 2 gives

(1-x) F(x) = R1 + (R2 – R1)x + (R3 – R2)x2 +

……………+ (Rn – Rn-1)xn-1+ ….. ∞

(Continued)

22.10.2005 11Prof. Pushpak Bhattacharyya, IIT Bombay

(1-x) F(x) = 2 + 2x + 3x2 + ……………+ nxn-1+ ….. ∞

F(x) = (1-x)-1(2 + 2x + 3x2 + ……………+ nxn-1+ ….. ∞)

= ( 1 + x + x2 + x3 +…+ xn-1+…..)

(2 + 2x + 3x2 + 4x3 +…+ nxn-1+…..)

(Continued)

22.10.2005 12Prof. Pushpak Bhattacharyya, IIT Bombay

Co eff of Xn-1 from the expression for f(x)

= ( n + n-1 + n-2 + … + 2+ 2 )

n(n+1)/ 2 + 1

(Continued)

22.10.2005 13Prof. Pushpak Bhattacharyya, IIT Bombay

Clustering Algos

1. Competitive learning

2. K – means clustering

3. Self organization / Kohonen net

4. Counter Propagation

22.10.2005 14Prof. Pushpak Bhattacharyya, IIT Bombay

Two packages

1. SNNS – Stuttgart Neural Net Simulator

2. WEKA Package

3. MATLAB

22.10.2005 15Prof. Pushpak Bhattacharyya, IIT Bombay

Data to train

1) IRIS data: 3 - class data

2) X- OR, parity, Majority

22.10.2005 16Prof. Pushpak Bhattacharyya, IIT Bombay

IRIS Data

Classification of flowers based on four attributes in 3 classes c1, c2 and C2

1 Petal length A1

2 Petal width A2

3 Sepal length A3

4 Sepal width A4

22.10.2005 17Prof. Pushpak Bhattacharyya, IIT Bombay

Google on “Machine learning Data”

IRIS

120 – data points

< A1, A2, A3, A4 >1 --- C1

< A1, A2, A3, A4 >2 --- C3

……….

……….

22.10.2005 18Prof. Pushpak Bhattacharyya, IIT Bombay

C1, C2, C3 each has 50 data points in it

80 – 20 rule for ML

Training – 40 and Testing -- 10

22.10.2005 19Prof. Pushpak Bhattacharyya, IIT Bombay

What you have to do for IRIS choose 40 points randomly from C1, C2 and C3

1. Run BP on FF n/w

2. Run SOM assuming unlabelled data. See if you can discover 3 cluster

22.10.2005 20Prof. Pushpak Bhattacharyya, IIT Bombay

Two part assignment

Supervised

Hidden layer to be found

4 neurons for 4 attributes

22.10.2005 21Prof. Pushpak Bhattacharyya, IIT Bombay

BP for IRIS error

Training iterations

EPOCHS

1%

Very good performance on test data should be achieved.

22.10.2005 22Prof. Pushpak Bhattacharyya, IIT Bombay

A1 A2 A3

4 I/p neuronsA4

Cluster Discovery By SOM/Kohenen Net

22.10.2005 23Prof. Pushpak Bhattacharyya, IIT Bombay

K – means clustering

K o/p neurons are required from the knowledge of k clusters being present.

26 neurons

Full connection

n neurons

……

……

22.10.2005 24Prof. Pushpak Bhattacharyya, IIT Bombay

Steps

1. Initialize the weights randomly.

2. Ik is the vector presented at kth iteration.

3. Find W* such that

|w* - Ik| < |wj - Ik| for all j

22.10.2005 25Prof. Pushpak Bhattacharyya, IIT Bombay

4. make W*(new) = W* (old) + η(Ik - w* ).

5. k K +1 ; go to 3.

6. Go to 2 until the error is below a threshold.

22.10.2005 26Prof. Pushpak Bhattacharyya, IIT Bombay

K means

1. Initialize weights randomly.

2. Ik in the I/p vector presented at kth.

step of one iteration over all the training data (EPOCH).

3. Find the W* such that |w* - Ik| is minimum.

4. Find W* for Ik+1..

22.10.2005 27Prof. Pushpak Bhattacharyya, IIT Bombay

5. See the partitions at the end of the epoch.

6. Find centroid of each partition.

7. Make the weight equal to the centroid of the partition.

8. Take next epoch.

22.10.2005 28Prof. Pushpak Bhattacharyya, IIT Bombay

9. Keep doing until the error is below a threshold.

Key idea form competitive learning: Weight change incorporate after every epoch.