Timo Honkela: An introduction to machine learning and neural networks

Timo Honkela, Modeling Meaning and Knowledge, 29.2.2016

Timo Honkela

Modeling Meaning and Knowledge29 Feb 2016

[email protected]

A history of machine learning and neural networks research


Learning as a basiccognitive process

● Learning helps animals to encounter their complex and changing environment

● In some cases, much of the learning takes place evolutionarily, whereas in some others, considerable proportion takes place through individual development

● Humans have a long childhood for intensive learning and creativity


Learning as a means to

● Cope in novel situations● Solve unseen problems● Build new knowledge

● Cf. planned economy: all needs shouldbe known beforehand

Coding knowledge by handhas its limitations


Carbonell, Michalski & Mitchell:

Machine Learning: A Historicaland Methodological Analysis (1983)

● Rote learning● Learning from instruction● Learning by analogy● Learning from examples● Learning from observation and discovery

http://www.aaai.org/ojs/index.php/aimagazine/article/view/406

http://dx.doi.org/10.1609/aimag.v4i3.406

http://dx.doi.org/10.1609/aimag.v4i3.406


Modeling learning:Nature or theory?

https://en.wikipedia.org/wiki/Wright_brothersPhoto by Timo Honkela


Learning as inductive inference

● Deduction: finding logically certain conclusion● Abduction: looking for explanation(s)● Induction: constructing generalization(s)

Ilkka NiiniluotoJaakko Hintikka Esa SaarinenNiiniluoto, I., & Tuomela, R. (2012). Theoretical concepts and hypothetico-inductive inference (Vol. 53). Springer.

Niiniluoto, I. (1972). Inductive systemati-zation: Definition and a critical survey.Synthese, 25(1-2), 25-81.

Hintikka, J. (1999). The concept ofinduction in the light of the interrogativeapproach to inquiry. In Inquiry asInquiry: A Logic of Scientific Discovery(pp. 161-181). Springer

Hintikka, J., & Suppes, P. (Eds.). (1966). Aspects of inductive logic (Vol. 49). North Holland

Saarinen, E. (1979). Backwards-looking operators in tense logicand in natural language. In Game-theoretical semantics (pp. 215-244). Springer.


Was Chomsky right?

● The Poverty of the Stimulus:Natural language grammar is unlearnable given data available to children

● Therefore that this knowledge is based on or supplemented with innate linguistic capacity

https://en.wikipedia.org/wiki/Poverty_of_the_stimulus


Cyc

● Douglas Lenat claimed that machine learning is possible only when substantial amount of knowledge is first coded to the system

● “In 1986, Doug Lenat estimated the effort to complete Cyc would be 250,000 rules and 350 man-years of effort.”

https://en.wikipedia.org/wiki/Cyc


Early history of neural networks

● “In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a paper on how neurons might work. In order to describe how neurons in the brain might work, they modeled a simple neural network using electrical circuits.”

● “In 1949, Donald Hebb wrote The Organization of Behavior, a work which pointed out the fact that neural pathways are strengthened each time they are used, a concept fundamentally essential to the ways in which humans learn. If two nerves fire at the same time, he argued, the connection between them is enhanced.”

● “In 1972, Kohonen and Anderson developed a similar network independently of one another [...] The neurons are supposed to activate a set of outputs instead of just one.”

https://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/History/history1.html


Learning paradigms

● Supervised learning● Semi-supervised learning● Reinforcement learning● Unsupervised learning


Self-Organizing Map

● The SOM by Teuvo Kohonen is the most popular neural network among unsupervised learning models

● Kohonen published the method in 1982● The SOM got its inspiration from what was

known about the cortical organization of the human brain

● Even nowadays the SOM explains well some basic cognitive and neural mechanisms


… and even hundresed of other books in which the Self-Organizing Map is a central topic.


http://wsom2016.rice.edu/files/2016/01/SOM-publications-stat-279p9hj.pdf


Honkela on Poverty of the stimulus

● Chomsky's point of view is narrow,influenced heavily by the formal framework

● Learning language includes bothimplicit and explicit feedback

● Much of the learned linguistic knowledgeis implicit in nature

● Linguistic categories such as noun and verb as implicit entities are learned from simple data in an unsupervised manner as shown e.g. by Honkela,Pulkka and Kohonen 1995

Timo Honkela: An introduction to machine learning and neural networks

Education

Transcript of Timo Honkela: An introduction to machine learning and neural networks