Artiﬁcial Neural Networks - Western Michigan University · 2016. 2. 4. · Introduction Concepts...

Introduction to Learning AlgorithmsHill Climbing

Simulated AnnealingSummary

Artificial Neural NetworksAn Introductory Look

Sayed Jahed Hussini & Hisham Saleh

Western Michigan UniversityDepartment of Computer ScienceAdvanced Theory of Computation

Dr. Elise de Doncker

February 4, 2016

Hussini & Saleh Artificial Neural Networks

IntroductionConcepts

ApplicationsCase Study

Benefits/LimitationsQuestions

Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks

Source: http://gizmodo.com/these-are-the-incredible-day-dreams-of-artificial-

neura-1712226908




ProblemSolution

Outline

1 IntroductionProblemSolution

2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture

3 Applications

4 Case StudyBankruptcy Prediction

5 Benefits/Limitations

6 Questions





ProblemSolution

Machines will follow a path that mirrors the evolutionof humans.

“Ray Kurzweil”





ProblemSolution

Problem

In 2012, Google received over 2 million search queries perminute

In 2014 it received over 4 million search queries per minute

Every Second:


Source: http://pennystocks.la/internet-in-real-time/




ProblemSolution

Problem

World is full of data

In today’s interconnected e-world, information can be storedand transmitted instantly

Challange?

To generate useful knowledge from collecteddata





ProblemSolution

Problem

Question How do we extract knowledge from noisy mass ofdata?

Traditional computers are too dumb tounderstand patterns or do analysis

Solution Empirical computer models that learn

Interpretation requires data acquisition, cleaning(preparing the data for analysis),Key is to extract information about data fromrelationships buried within the data itself.





ProblemSolution

Solution

Human Brain is the most powerful computer every invented to doanalysis

However it cannot handle huge amounts of data





ProblemSolution

Artificial Neural Networks(ANN)

Rather than programming computers to the most specificdetail of their tasks teach them how to do a job e.g: children

We must built empirical models that can find patterns rapidlyand accurately(to some extent) burried in data

Artificial Intelligence System - AI can do this

ANN is a case of AI





Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture

Outline



3 Applications



6 Questions






Artificial Neural Networks(ANN)

ANN is an information processing paradigm that is inspired bythe way biological nervous systems, such as the brain, processinformation

The key element in this paradigm is the novel structure ofinformation processing

ANNs, like people, learn by example

Currently, an ANN is configured for a specific application e.g:pattern recognition, data calssification






How human brain learns?

Human brain is a dense network of approximately 1011

neurons, each connected to, on average, 104 others

Neuron activity is excited or inhibited through connections toother neurons






From Natural to Artificial Neurons

To build artificial neuron:

Deduce essential features of neurons and their connections

Program a system to simulate the features

Due to imprecise knowledge, our models are necessarily grossidealisations of real networks of neurones






A simple neuron

Artificial neuron is a device with many inputs and one output

Two modes:

TrainingUsing

Firing Rule determines when a neuron should fire.

Are very important in neural networks and accounts for theirhigh flexibility

Calcualtions of when neuron should fire are based on inputpatterns






A simple neuron






Perceptrons

A type of atrificial neuron developed in 1905s

Takes several binary inputs and produces a single ouput

To compute the output each input is given a weight, thatexpresses it’s importance

The output is determined:

output =

{0 if

∑j wjxj ≤ threshhold

1 if∑

j wjxj ≥ threshhold


Source:http://neuralnetworksanddeeplearning.com/chap1.html





Preceptrons

Example: Decide whether to go to a festival or not:

How is the weather?(x1)How far is the festival grounds?(x2)Does your boyfriend/girlfriend want to accompany you?(x3)

A complex perceptron:







Sigmoid

The problem is that this isn’t what happens when our networkcontains perceptrons

In fact, a small change in the weights or bias of any singleperceptron in the network can sometimes cause the output ofthat perceptron to completely flip, say from 00 to 1







Sigmoid

The aforementioned problem is solved by another type ofartificial neuron called Sigmoid neuron

Similar to perceptrons, but modified so that small changes intheir weights and bias cause only a small change in theiroutput

In Sigmoid neurons inputs instead of just being 0 and 1, cantake any value between 0 and 1






Sigmoid







Architecture

Feed-Forward Networks Backpropagation Networks



andhttp://nikhilbuduma.com/2015/01/11/a-deep-dive-into-recurrent-neural-networks/





ANN Application Development

Nelson and Illingworth in ”A Practical Guide To Neural Networks”outline following steps on designing a neural network:

1 Variable selection

2 Data collection

3 Training, testing and validation set

4 Network Architecture

Number of hidden layers and neuronsNumber of ouput neuronsTransfer function

5 Neural Network Training

6 implementation





Outline



3 Applications



6 Questions





Applications

Security(e.g: Baggagechecking in airports)

Stock market prediction

Loan approval

Credit rating

Medical diagnosis

Process/Quality control

Pattern recognition

Recognizing genes

Ecosystem evaluation

Kndowledge discovery

Time serie analysis

Sales forecasting

Targetted marketing

HR management





Bankruptcy Prediction

Outline



3 Applications



6 Questions






Problem Statement

There have been a lot of work on developing neural networksto predict bankruptcy using financial ratios and discriminantanalysis

The ANN paradigm selected in the design phase for thisproblem was a three-layer feedforward ANN usingbackpropagation

The data for training the network consisted of a small set ofnumbers for well-known financial ratios, and data wereavailable on the bankruptcy outcomes corresponding to knowndata sets






Application Design

There are five input nodes, corresponding to five financial ratios:

X1: Working capital/total assets

X2: Retained earnings/total assets

X3: Earnings before interest and taxes/total assets

X4: Market value of equity/total debt

X5: Sales/total assets

A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firmThe system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past






Application Design







A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firm

The system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past






Application Design







A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firmThe system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past






ANN Architecture


Source:www.cse.hcmut.edu.vn/ dtanh/download/ANN.ppt





Training and Testing

Training:

The data set, consisting of 129 firms, was partitioned into atraining set and a test set. The training set of 74 firmsconsisted of 38 that went bankrupt and 36 that did not

Testing:

The test set consisted of 27 bankrupt and 28 non-bankruptfirms. The neural network was able to correctly predict 81.5%of the bankrupt cases and 82.1% of the nonbankrupt casesOverrall, the ANN did much better predicting 22 out of the27 actual cases (the discriminant analysis predicted only 16cases correctly)

Source: R.L. Wilson and R. Sharda, “Bankruptcy Prediction UsingNeural Networks,” Decision Support Systems, Vol. 11, No. 5, June1994, pp. 545-557.





Outline



3 Applications



6 Questions





Benefits

Useful in pattern recognition, classification, abstraction andinterpretation of incomplete and noisy inputs e.g:handwritting

Providing some human characteristics to problem solving thatare difficult using standard system/software

Ability to solve new kinds of problems. ANNs are particularlyeffective at solving problems whose solutions are difficult, ifnot impossible, to define

ANNs tend to be more robust, and have the ability to copewith imcomplete or fuzzy data.





Limitations

ANNs do not produce an explicit model even though newcases can be fed into it and new results obtained

ANNs lack explanation capabilities. Justifications for results isdifficults to obtain because the connection weights usually donot have obvious interpretaions





Google’s DeepDream


Source: http://gizmodo.com/these-are-the-incredible-day-dreams-of-artificial-

neura-1712226908




Outline



3 Applications



6 Questions





Questions

Questions?





References

Uhrig, R.E., ”Introduction to artificial neural networks,” inIndustrial Electronics, Control, and Instrumentation, 1995.,Proceedings of the 1995 IEEE IECON 21st InternationalConference on , vol.1, no., pp.33-37 vol.1, 6-10 Nov 1995Introduction to artificial neural networks,” in ElectronicTechnology Directions to the Year 2000, 1995. Proceedings. ,vol., no., pp.36-62, 23-25 May 1995Yuhong Li; Weihua Ma, ”Applications of Artificial NeuralNetworks in Financial Economics: A Survey,” inComputational Intelligence and Design (ISCID), 2010International Symposium on , vol.1, no., pp.211-214, 29-31Oct. 2010Huang, S.H.; Hong-Chao Zhang, ”Artificial neural networks inmanufacturing: concepts, applications, and perspectives,” inComponents, Packaging, and Manufacturing Technology, PartA, IEEE Transactions on , vol.17, no.2, pp.212-228, Jun 1994





References

https://www.doc.ic.ac.uk/ nd/surprise96/journal/vol4/cs11/report.htmlhttp ://neuralnetworksanddeeplearning .com/chap1.html

http://nikhilbuduma.com/2015/01/11/a-deep-dive-into-recurrent-neural-networks/

http://natureofcode.com/book/chapter-10-neural-networks/

http://www.theatlantic.com/technology/archive/2015/09/robots-hallucinate-dream/403498/

https://www.technologyreview.com/s/513696/deep-learning/

uhaweb.hartford.edu/ilumokanw/Chap1student.ppt

www.cse.hcmut.edu.vn/ dtanh/download/ANN.ppt




Artificial Neural NetworksAn Introductory Look

Sayed Jahed Hussini & Hisham Saleh

Western Michigan UniversityDepartment of Computer ScienceAdvanced Theory of Computation

Dr. Elise de Doncker

February 4, 2016




Outline

1 Introduction to Learning AlgorithmsWhat Are They?

2 Hill ClimbingThe algorithmDisadvantages

3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting




What Are They?

Outline







What Are They?

What Are Learning Algorithmsand How do they work

As Jahed discussed in his presentation, neural networksbecome accurate as they are trained more.Training a neural network and building its neuronconnections requires a set of algorithms that fall within therealm of machine learning. These algorithms are generalartificial intelligence algorithms that can applied to helptrain the neuron network.By the end of this presentation, it is my hope that you willhave an additional two algorithms to add to your list ofthings that confuse you.




What Are They?

The Algorithms

Hill Climbing:The first algorithm we will cover is the hill climbingalgorithm, a technique that allows you to conduct a "local"search. This is one of the simplest technique available inthe artificial

Simulated Annealing:The algorithm is used to allow us to approximate theoptimal solution to a problem with too many possiblesolutions to reasonably consider all of them in the search.Simulated Annealing is an algorithm thats based on thesimilar annealing process in metallurgy. We will cover how itworks and its advantages in a few slides.




The algorithmDisadvantages

Outline








The Algorithm

The basic idea in Hill Climbing is that the solution or goalstate is at the top of the highest hill and you must reach it.Algorithm:

First generate an initial solution.Loop till the crest is reached.

Check neighboring pointIf it is better, choose it as solutionOtherwise, you’ve reached the crest





Example

Figure: Image from http://www35.homepage.villanova.edu/abdo.achkar/csc8530/proj.htm


http://www35.homepage.villanova.edu/abdo.achkar/csc8530/proj.htm

http://www35.homepage.villanova.edu/abdo.achkar/csc8530/proj.htm




Outline








Local Maxima/Minima

Figure: Image from http://webspace.ulbsibiu.ro/adrian.florea/html/Planificari/EvolutionaryComputing/Course_3/ppt/Hill_Climbing.ppt


http://webspace.ulbsibiu.ro/adrian.florea/html/Planificari/EvolutionaryComputing/Course_3/ppt/Hill_Climbing.ppt






Plateau

Figure: Image from http://webspace.ulbsibiu.ro/adrian.florea/html/Planificari/EvolutionaryComputing/Course_3/ppt/Hill_Climbing.ppt








Getting out?

How do we get out when we’re stuck?Backtracking to some earlier node and choosing a differentpath.Making a big jumpOthers?




DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting

Outline








Definition

Simulated Annealing works by trying to achieve a goalstate without reaching it too fast. What?In search algorithms, we want to focus on solutions thatmight be optimal without ignoring better solutions that wemight end up finding later. We want to make sure we don’tget stuck in the local optimal solutionAnnealing in metallurgy is a process that applies heat to asubstance in order to alter its physical properties in orderto increase its malleability and decrease its hardness.What does that have to do with anything?





Outline








The Algorithm

1 First thing is to pick a solution. What is its2 while temperature is greater than minimum temperature

"energy value"?1 Store a copy of the current solution2 Change the copy slightly.3 Compare the new energy value with the old energy value.

and keep the better one.4 Reduce the temperature.

3 Repeat all the above (Why?)(Where do we stop?)





Why does this work

In the beginning, we create a variable to represent thetemperature. We set the temperature to be high.When the temperature is high, we allow the algorithm willbe allowed to more often accept solutions that are not asgood as the one we have right now. (Why?)Reduction of the temperature allows the algorithm toreduce its acceptance of worse solutions, thus allowing itto focus on an area of the search space.Simulated Annealing allows the algorithm to takeadvantage of the fact that a solution is easy to find.





Outline








Advantages

Because simulated annealing can accept worse solutionsat times, it does not get stuck at local optima as much asthe hill climbing technique.In general, simulated annealing is much better atapproximating the global optimal solution. It is muchsimpler than the more complicated genetic techniques yetvery powerful





Outline








Acceptance Function

How do we determine how to accept the solution and stop?Well there’s an equation for that (sort of). We mustdetermine whether the new solution is better than ourcurrent one, or if it’s worse, by how much.T̂hat leads us to the question of how do we quantify worse?The math is pretty simple: exp( (current Energy -neighbor’s Energy) / temperature) Basically, the smaller thechange in energy and the higher the temperature, the morelikely the new solution will be accepted.





Setting the parameters

1 Initial temperature: The temperature should be chosen ashigh as possible so that initially, the current solution will beaccepted.

2 Temperature change: Setting the cooling rate too high willforce the algorithm into a smaller region without first takinga glance at the entire search space.

3 Minimum Temperature: Setting the temperature too low willforce the algorithm to search in a small region for too long,rather than allowing it to escape from the local area.

4 Acceptance delta: At what point do you simply accept thesolution you have already as a close enoughapproximation of the optimal solution?





Effect of Initial Solution

Question: How much does your choice of the initial solutionaffect your final result?

What about in comparison to hill climbing?




Summary

Neural Networks should be in every computer scientist’stoolbox.Hill climbing and Simulated Annealing are just twomethods used to teach a neural network.




Questions

1 How do we extract knowledge from noisy mass of data?2 Explain the difference between Sigmoids and Perceptrons.

Describe when do we use one over the other?3 Describe some of the benefits of Artificial Neural Networks.4 Describe some methods of getting out when the hill

climbing algorithm gets stuck.5 Describe the simulated annealing algorithm and explain its

advantages over hill climbing.6 Explain some of the parameters that must be set for

simulated annealing to work.


Artiﬁcial Neural Networks - Western Michigan University · 2016. 2. 4. · Introduction Concepts...

Documents

Transcript of Artiﬁcial Neural Networks - Western Michigan University · 2016. 2. 4. · Introduction Concepts...