Neural Networks in Computer Science

CS/PY 231 Lab Presentation # 1 January 14, 2005 Mount Union College

Some Computing History

First all-electronic computing device: ENIAC (1946) Ekert & Mauchly (Penn)– application: computing ballistic firing tables

for US Navy– special-purpose device: hard-wired, like

the computer in your car’s engine• to change the problem, device must be rewired• all memory dedicated to data storage

Innovation: Stored Program Concept First stored program computer: EDSAC

(1952) von Neumann (Princeton) reprogramming became a logical (not

physical) task memory stores both software and data much easier to program; therefore, use

of computers explodes commercially

von Neumann Architecture

one powerful, fast processor– device capable of performing calculations

and comparisons large memory that the processor may

access serial processing model: processor

performs one operation at a time, in series

von Neumann Architecture

Memory

ProcessorProgram Instructions

Data

Traits of von Neumann Machines

much better than humans at performing calculations and logical step-by-step procedures

since computers were so much better at these kinds of tasks, it was assumed that they would outperform humans in all mental tasks– as soon as we could determine how to

program the machines for those tasks

Not So Fast, My Friend…

von Neumann architecture turns out to have serious limitations when confronted with certain problems

von Neumann bottleneck: even with a fast processor, doing one thing at a time is too slow when facing a huge task

idea: have several processors working on the same problem

Example: Sorting a list of values

data: 45, 67, 12, 87, 19, 55, 32, 29 method: bubble sort single processor: 27 comparisons, 16

switches two processors: 12 comparisons, 5

switches, 8 merge steps complications: coordination of processors,

communication, limited # of processors, etc.

A Paradox

Researchers discovered that machines based on the von Neumann architecture are:– good at problems that are difficult for humans

(e.g., calculations), and– bad at problems that humans find to be easy

(e.g., pattern recognition) This seems to be a limitation of the

machine’s architecture

A Novel Idea

Why not build computers that are structured as brains are?

Problem: brain structure is fabulously complex and consists of massive numbers of processors (neurons)

We can start with small artificial brains and add complexity as we progress

Processing Speeds

Modern processors are almost a billion times faster than real neurons

However, the brain contains hundreds of billions of neurons, with trillions of connections, each of which can process and store signals

The brain ends up having much more raw processing power than a single processor

Essential for Pattern Recognition

Massive Parallelism many tasks are performed by the brain at

the same instant of time (breathing control, digestion, visual processing, thinking, etc.)

problem: how to program such a complex combination of processing devices?

solution: look to nature!

Neural Networks

term for a computing device based on brain structure

idea: connecting a huge number of simple processors into a large interconnected network

organized in layers: input, middle (hidden), output

How does this network know what answers to produce for some input?

Traditional Computer Programming we must develop a step-by-step

sequence of operations that when carried out will produce the answers to the problem being considered

someone must write such a program -- this is a difficult task

writing software for parallel architectures is MUCH harder than this!!

Training Neural Networks

A neural network may be trained to produce a desired output when exposed to a given input pattern

it turns out that networks are GREAT at solving pattern recognition tasks

they may be trained to recognize any patterns (visual, audio, representational)– as long as the input is encoded in a form that the

network can receive)

Modeling Nature

much of the complexity of a brain cell is present to support the living operation of the cell

we can devise a simplified model of the neuron that focuses on the processing function of the cell

we will then use these units to construct artificial neural networks (brains)

The Perceptron

a simple model of a real neuron contains essential features:

– a number of input connections (values received from other neurons)

– each input connection has an associated weight

– the perceptron has a threshold value, and produces

– an output value (the answer)

The Perceptron

I1

I2

I3

In

: :

Output

(threshold)

w1

w2

w3

wn

The Perceptron’s Calculation

at any instant of time, the output generated by the perceptron is calculated as follows:

first, each input signal is multiplied by the weight associated with that connection

all of these products are added together if the sum equals or exceeds the threshold,

the perceptron “fires”

The Perceptron’s Calculation

if the sum is less than the threshold value, the perceptron does not produce a signal

we use output values of 1 for firing and 0 for not firing

simplification of what actual brain cells do– but this simple concept is the basis for neural

network computing!

Simplest Perceptron

limited to two input connections can be used to classify input patterns

into classes (accepted or rejected) Perceptrons can be used to solve a

wide variety of decision problems– but are too simple for many other problems– for those, we’ll have to combine

perceptrons in a hierarchy or network

Neural Networks in Computer Science

CS/PY 231 Lab Presentation # 1 January 14, 2005 Mount Union College