CSC 4510 – Machine LearningDr. Mary-Angela PapalaskariDepartment of Computing SciencesVillanova University

Course website: www.csc.villanova.edu/~map/4510/

7: Introduction to Neural Networks

1CSC 4510 - M.A. Papalaskari - Villanova University

Some of the slides in this presentation are adapted from:• Prof. Frank Klassner’s ML class at Villanova• the University of Manchester ML course http://www.cs.manchester.ac.uk/ugt/COMP24111/• The Stanford online ML course http://www.ml-class.org/

Machine learning problems• Supervised Learning

– Classification– Regression

• Unsupervised learning

Others: Reinforcement learning, recommender systems.

Also talk about: Practical advice for applying learning algorithms.

CSC 4510 - M.A. Papalaskari - Villanova University 2

Motivation – part 1• Learning a non-linear function

CSC 4510 - M.A. Papalaskari - Villanova University 3

You see this:

But the camera sees this:

What is this?

Computer Vision: Car detection

Testing:

What is this?

Not a carCars

Learning Algorithm

pixel 1

pixel 2

pixel 1

pixel 2

Raw image

Cars“Non”-Cars

pixel 1

pixel 2

Raw image

Cars“Non”-Cars

Learning Algorithm

pixel 1

pixel 2

pixel 1

pixel 2

Raw image

Cars“Non”-Cars

Learning Algorithm

pixel 1

pixel 2

Why not apply logistic regression?

Non-linear decision boundaries

x1

x2

Predict “ “ if

1-1

-1

1

Reminder: logistic regression can do non-linear

Add some higher-order terms?

pixel 1

pixel 2

Raw image

Cars“Non”-Cars

50 x 50 pixel images→ 2500 pixels (7500 if RGB)

pixel 1 intensity

pixel 2 intensity

pixel 2500 intensity

Quadratic features ( ): ≈3 million features

Learning Algorithm

pixel 1

pixel 2

Motivation – part 2• Take inspiration from the brain

CSC 4510 - M.A. Papalaskari - Villanova University 11

Neural Networks

Origins: Algorithms that try to mimic the brain.

• Was very widely used in 80s and early 90s;• popularity diminished in late 90s.• Recent resurgence: State-of-the-art technique for

many applications

Auditory cortex learns to see

Auditory Cortex

The “one learning algorithm” hypothesis

[Roe et al., 1992]

Somatosensory cortex learns to see

Somatosensory Cortex

The “one learning algorithm” hypothesis

[Metin & Frost, 1989]

Seeing with your tongue Human echolocation (sonar)

Haptic belt: Direction sense Implanting a 3rd eye

Sensor representations in the brain

[BrainPort; Welsh & Blasch, 1997; Nagel et al., 2005; Constantine-Paton & Law, 2009]

Neuron in the brain

Input signals sentfrom other neurons

If enough sufficient signals accumulate, the neuron fires

a signal.

Connection strengths determine how the signals are accumulated

Neurons in the brain

[Credit: US National Institutes of Health, National Institute on Aging]

Comparing Carbon & SiliconHuman Brain

Computational Units: O(1011) neuronsStorage Units: O(1011) neurons, O(1014) synapsesCycle Time: O(10-3) sec, Bandwidth: O(1014) bits/secNeuron Updates/sec: O(1014)

ComputerComputational Units: 1 CPU, 107 gatesStorage Units: O(1011) bits RAM, O(1012) bits diskCycle Time: O(10-8) sec, Bandwidth: O(108) bits/secNeuron Updates/sec: O(108)

The neuron modelMcCulloch & Pitts 1943

• Compute weighted sum of inputs and “fire” if above threshold value

CSC 4510 - M.A. Papalaskari - Villanova University 19

1x

2x

3x

add

)( taif 1output output

signal

• input signals ‘x’ and coefficients ‘w’ are multiplied

• weights correspond to connection strengths

• signals are added up – if they are enough, FIRE!

else0output

1w

2w

3w

i

M

iiwxa

1

incoming signal

connection strength

activationlevel

output signal

Activation Functions

A neuron can compute….

The perceptronRosenblatt 1958: Training algorithm Neural Net• A single neuron• Adjustable synaptic weights

CSC 4510 - M.A. Papalaskari - Villanova University 23

Perceptron learning1. Initialize weights and thresholds to random

numbers between -0.5 and 0.52. Activate perceptron3. Update weights:wi(p+1) = wi(p) + α * xi(p) * err(p)

4. Iterate until Convergence

Example: Perceptron

learning logical AND

25

Representation Limits for perceptron

Linear Separability