COMP5331 1

COMP5331

Other Classification Models:Neural Network

Prepared by Raymond WongSome of the notes about Neural Network are borrowed from LW Chan’s notes

Presented by Raymond Wongraywong@cse

COMP5331 2

What we learnt for Classification Decision Tree Bayesian Classifier Nearest Neighbor Classifier

COMP5331 3

Other Classification Models Neural Network

COMP5331 4

Neural Network Neural Network

A computing system made up of simple and highly interconnected processing elements

Other terminologies: Connectionist Models Parallel distributed processing models (PDP) Artificial Neural Networks Computational Neural Networks Neurocomputers

COMP5331 5

Neural Network This approach is inspired by the way

that brains process information, which is entirely differ from the way that conventional computers do

Information processing occurs at many identical and simple processing elements called neurons (or also called units, cells or nodes)

Interneuron connection strengths known as synaptic weights are used to store the knowledge

COMP5331 6

Advantages of Neural Network Parallel Processing – each

neuron operates individually Fault tolerance – if a small

number of neurons break down, the whole system is still able to operate with slight degradation in performance.

COMP5331 7

Neuron Network Neuron Network for OR

Neuron Network

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

COMP5331 8

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Neuron

COMP5331 9

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Front Backnet

COMP5331 10

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Front Backnetw1

w2

Weight net = w1x1 + w2x2 + b

COMP5331 11

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Front Backnetw1

w2

Activation function-Linear function: y = net or y = a . net-Non-linear function

COMP5331 12

Activation Function Non-linear functions

Threshold function, Step Function, Hard Limiter

Piecewise-Linear Function Sigmoid Function Radial Basis Function

COMP5331 13

Threshold function, Step Function, Hard Limiter

y = if net 0

if net <0

1

0

0 net

y

1

COMP5331 14

Piecewise-Linear Function

y =

if net a

if –a < net < a

1

0

0 net

y

1

if net -a

½(1/a x net + 1)

-a a

COMP5331 15

Sigmoid Function

y =

0 net

y

1

11 + e-net

COMP5331 16

Radial Basis Function

y =

0 net

y

1

-net2e

COMP5331 17

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Front Backnetw1

w2

net = w1x1 + w2x2 + b

Threshold function

y = if net 0

if net <0

1

0

COMP5331 18

Learning Let be the learning rate (a real

number) Learning is done by

wi wi + (d – y)xi where

d is the desired output y is the output of our neural network

b b + (d – y)

COMP5331 19

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 d0 0 00 1 11 0 11 1 1

OR Function

Front Backnetw1

w2

net = w1x1 + w2x2 + b

Threshold function

y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

COMP5331 20

Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b

b w1 w21 1 1

=1

y = 1

w1 = w1 + (d – y)x1

= 1+0.8*(0-1)*0 = 1

w2 = w2 + (d – y)x2

= 1+0.8*(0-1)*0 = 1

b = b + (d – y)

= 1+0.8*(0-1) = 0.2

0.2 1 1

Incorrect!

= 0.8

COMP5331 21

Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b

b w1 w20.2 1 1

=1.2

y = 1

w1 = w1 + (d – y)x1

= 1+0.8*(1-1)*0 = 1

w2 = w2 + (d – y)x2

= 1+0.8*(1-1)*1 = 1

b = b + (d – y)

= 0.2+0.8*(1-1) = 0.2

0.2 1 1

Correct!

= 0.8

COMP5331 22

Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b

b w1 w20.2 1 1

=1.2

y = 1

w1 = w1 + (d – y)x1

= 1+0.8*(1-1)*1 = 1

w2 = w2 + (d – y)x2

= 1+0.8*(1-1)*0 = 1

b = b + (d – y)

= 0.2+0.8*(1-1) = 0.2

0.2 1 1

Correct!

= 0.8

COMP5331 23

Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b

b w1 w20.2 1 1

=2.2

y = 1

w1 = w1 + (d – y)x1

= 1+0.8*(1-1)*1 = 1

w2 = w2 + (d – y)x2

= 1+0.8*(1-1)*1 = 1

b = b + (d – y)

= 0.2+0.8*(1-1) = 0.2

0.2 1 1

Correct!

= 0.8

COMP5331 24

Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1

net = w1x1 + w2x2 + b y = if net 0

if net <0

1

0

net = w1x1 + w2x2 + b

b w1 w20.2 1 1

=0.2

y = 1

w1 = w1 + (d – y)x1

= 1+0.8*(0-1)*0 = 1

w2 = w2 + (d – y)x2

= 1+0.8*(0-1)*0 = 1

b = b + (d – y)

= 0.2+0.8*(0-1) = -0.6

-0.6 1 1

Incorrect!

We repeat the above process until the neural networks output the correct values of y (i.e., y = d for each possible input)

= 0.8

COMP5331 25

Neuron Network Neuron Network for OR

input

outputx1

x2y

x1 x2 y0 0 00 1 11 0 11 1 1

OR Function

Front Backnetw1

w2

net = w1x1 + w2x2 + b

Threshold function

y = if net 0

if net <0

1

0

COMP5331 26

Limitation It can only solve linearly separable

problems

COMP5331 27

Multi-layer Perceptron (MLP)

Neuron Network

input

outputx1

x2y

input

outputx1

x2yNeuron

COMP5331 28

Multi-layer Perceptron (MLP)

Neuron Network

input

outputx1

x2y

inputoutputx1

x2

y

COMP5331 29

Multi-layer Perceptron (MLP)

inputoutput

x1y1

x2

x3

x4

x5

y2

y3

y4

Input layer Hidden layer Output layer

COMP5331 30

Advantages of MLP Can solve

Linear separable problems Non-linear separable problems

A universal approximator MLP has proven to be a universal

approximator, i.e., it can model all types function y = f(x)