OutlineOutline What Neural Networks are and why they are

desirable Historical background Applications Strengths neural networks and advantages Status N.N and evolving N.N Basic Architectures and algorithms Applications Drawbacks and limitations

What are Neural Networks?What are Neural Networks?

information processing paradigm inspired by biological nervous systems, such as our brain

Structure: large number of highly interconnected processing elements (neurons) working together

Like people, they learn from experience (by example)

Historical backgroundHistorical background

Neural network simulations appear to be a recent development. However, this field was established before the advent of computers, and has survived at least one major setback and several eras.

These pioneers were able to develop convincing technology which surpassed the limitations identified by Minsky and Papert. Minsky and Papert, published a book (in 1969)

Human Brain FunctionHuman Brain Function

Neural Network FunctionNeural Network Function

TerminologyTerminologyInput: Explanatory variables also referred

to as “predictors”.Neuron: Individual units in the hidden

layer(s) of a neural network.Output: Response variables also called

“predictions”.Hidden Layers: Layers between input and

output that an apply activation function.

Some SimilaritiesSome Similarities

Why Neural Networks are desirableWhy Neural Networks are desirable

Human brain can generalize from abstract

Recognize patterns in the presence of noise

Recall memoriesMake decisions for current problems

based on prior experience

When to use neural networksWhen to use neural networks

Use for huge data sets (i.e. 50 predictors and 15,000 observations) with unknown distributions

Smaller data sets with outliers as neural networks are very resistant to outliers

Why Neural Networks in Statistics?Why Neural Networks in Statistics?

The methodology is seen as a new paradigm for data analysis where models are not explicitly stated but rather implicitly defined by the network.

Allows for analysis where traditional methods might be extremely tedious or nearly impossible to interpret.

Difference in Neural NetworksDifference in Neural Networks

The difference in the two approaches is that multiple linear regression has a closed form solution for the coefficients, while neural networks use an iterative process.

ApplicationsApplications Data Conceptualization– infer grouping relationships

e.g. extract from a database the names of those most likely to buy a particular product.

Data Filtering– e.g. take the noise out of a telephone signal,

signal smoothing Planning– Unknown environments– Sensor data is noisy– Fairly new approach to planning

ApplicationsApplications

Prediction: learning from past experience– pick the best stocks in the market– predict weather– identify people with cancer risk

Classification– Image processing– Predict bankruptcy for credit card

companies– Risk assessment

Strengths of a Neural NetworkStrengths of a Neural Network

Power: Model complex functions, nonlinearity built into the network

Ease of use:–Learn by example–Very little user domain-specific

expertise neededIntuitively appealing: based on model

of biology, will it lead to genuinely intelligent computers/robots?

General AdvantagesGeneral Advantages

Advantages– Adapt to unknown situations– Robustness: fault tolerance due to

network redundancy– Autonomous learning and generalization

Disadvantages– Not exact– Large complexity of the network structure

For motion planning?

Status of Neural NetworksStatus of Neural Networks

Most of the reported applications are still in research stage

No formal proofs, but they seem to have useful applications that work

Evolving networksEvolving networks

Continuous process of:– Evaluate output – Adapt weights– Take new inputs

ANN evolving causes stable state of the weights, but neurons continue working: network has ‘learned’ dealing with the problem

Where are NN used?Where are NN used?

Recognizing and matching complicated, vague, or incomplete patterns

Data is unreliable Problems with noisy data

–Prediction–Classification–Data association–Data conceptualization–Planning

Back propagationBack propagation

Desired output of the training examples

Error = difference between actual & desired output

Change weight relative to error sizeCalculate output layer error , then

propagate back to previous layerImproved performance, very common!

Activation FunctionActivation Function

The only practical requirement for an activation function is that it be differentiable

Sigmoid function is commonly used g(netinput) = 1/(1+ exp-(netinput))Or a simple binary threshold unitӨ(netinput) = {1 ,if netinput ≥ 0 ; 0 ,

otherwise}

Training the NetworkTraining the Network

Neural Networks must be first trained before being used to analyze new data

Process entails running patterns through the network until the network has “learned” the model to apply to future data

Can take a long time for noisy data

New DataNew Data

Once the network is trained new data can be run through it

The network will classify new data based on the previous data it trained with

If an exact match can not be found it will match with the closest found in memory

Regression and Neural NetworksRegression and Neural Networks

Objective of regression problem is to find coefficients that minimize sum of errors

To find coefficients we must have a dataset that includes the independent variable and associated values of the dependent variable. (very similar to training the network)

Equivalent to a single layer feed forward network

Drawbacks and LimitationsDrawbacks and Limitations

Neural Networks can be extremely hard to use

The programs are filled with settings you must input and a small error will cause your predictions to have error also

The results can be very hard to interpret as well