Artificial Neural Networks: Recent Advances, Current...
Transcript of Artificial Neural Networks: Recent Advances, Current...
![Page 1: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/1.jpg)
Eduardo Bezerra
09/mar/2018
Artificial Neural Networks:
Recent Advances, Current Trends
and Open Problems
![Page 2: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/2.jpg)
Roadmap
◻ Recent Advances
◻ Current Trends
◻ Open Problems
2
![Page 3: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/3.jpg)
Recent Advances 3
![Page 4: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/4.jpg)
Backpropagation
1986
4
![Page 5: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/5.jpg)
Universal Approximation Theorem
“We show that standard multilayer feedfbrward networks
with as few as a single hidden layer and arbitrary
bounded and nonconstant activation function are
universal approximators”
1991
5
![Page 6: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/6.jpg)
LSTM Neural Nets
https://scholar.google.de
1997 Citações: 2012: 77; 2016: 2133; 2017: 3932
“if you can understand the
paper, you are better than
many people in ML. It took 10
years until people understand
what they were talking about”.
– Jeff Hinton
Juergen Schmidhuber
Sepp Hochreiter
6
![Page 7: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/7.jpg)
LeNet
1998
Yann Lecun 7
![Page 8: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/8.jpg)
Neural Nets Renaissance
2000s
2009
2006
2007
2009
2009
8
![Page 9: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/9.jpg)
Deep Learning Explosion
Also known as: Revenge of the Sith Neural Nets! Also known as: Revenge of the Sith Neural Nets! 2010s
9
![Page 10: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/10.jpg)
Basic Idea: Hierarchy of Features
Credits: Andrew Ng apud The Learning Machines (Nature, 2014)
Composition of functions across layers
is a cornerstone of deep neural nets.
10
![Page 11: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/11.jpg)
Speech Recognition
2012
11
![Page 12: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/12.jpg)
12
Object Recognition (AlexNet)
2012 dropout
rectified linear activation units (ReLU)
8 layers
AlexNet
12
![Page 13: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/13.jpg)
Credits: Mathew Zeiler (Clarifai)
Object Recognition (AlexNet)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012
13
![Page 14: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/14.jpg)
Deep Nets
14
152 layers
22 layers
GoogleNet
ResNet
19 layers
VGGNet
14
![Page 15: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/15.jpg)
Deep Nets
15 Credits: https://medium.com/@Lidinwise/the-revolution-of-depth-facf174924f5
15
![Page 16: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/16.jpg)
Semantic Segmentation
16 2014
16
![Page 17: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/17.jpg)
Generative Adversarial Nets
2014
17
![Page 18: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/18.jpg)
Deep Reinforcement Learning
2015
18
![Page 19: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/19.jpg)
Deep Reinforcement Learning
2016
19
![Page 20: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/20.jpg)
Success Factors
More
computer
power
(GPUs)
More Data
Available
Tons
of $
New training
techniques &
methods
DEEP
LEARNING
Adapted from Dallagnol (2016). 20
20
![Page 21: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/21.jpg)
Success Factors
Geoffrey Hinton
“What was wrong in the 80’s is that
we didn’t have enough data and
we didn’t have enough computer
power”
21
![Page 22: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/22.jpg)
The Connectionist Invasion 22
![Page 23: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/23.jpg)
Current Trends 23
![Page 24: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/24.jpg)
Multimodal Learning
A Bimodal Learning Approach to Assist Multi-sensory Effects Synchronization
R. Abreu, J. Santos, E. Bezerra, submited to IJCNN2018 24
24
![Page 25: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/25.jpg)
Multimodal Learning
Long-term Recurrent Convolutional Networks for Visual Recognition and Description, 2016.
25
![Page 26: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/26.jpg)
Attention Models 26
![Page 27: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/27.jpg)
Learning to Learn (Meta learning)
Welcome to the Hyperparameter Jungle! ⬜ Optimization technique
⬜ Amount of epochs
⬜ Amount of layers, hidden units
⬜ Type of each layer, type of activation function
⬜ Cost function, regularization technique,
⬜ Learning rate, momentum
⬜ Early stopping, weight decay, minibatch size
⬜ ....
Y. Bengio. Practical Recommendations for Gradient-Based Training of Deep Architectures, 2012
27
![Page 28: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/28.jpg)
Learning to Learn (Meta learning)
◻ Current:
ML expertise + data + computation
◻ Future:
data + 100X computation
Credits: https://becominghuman.ai/jeff-deans-talk-on-large-scale-deep-learning-171fb8c8ac57
28
![Page 29: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/29.jpg)
Use of Prior Knowledge
◻ Error curve is flatening add prior knowledge! ⬜ Visual common-sense for scene understanding using
perception, semantic parsing and reasoning, S Aditya, Y Yang,
C Baral, C Fermuller, Y Aloimonos - 2015.
⬜ Distilling the knowledge in a neural network. Hinton G, Vinyals
O, Dean J. 2015.
⬜ Harnessing deep neural networks with logic rules. Hu Z, Ma X,
Liu Z, Hovy E, Xing E. 2016.
29
![Page 30: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/30.jpg)
Open Problems 30
![Page 31: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/31.jpg)
Is Winter Coming Again?!
Credits: Monty Barlow
31
![Page 32: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/32.jpg)
Unsupervised Learning
Current models are hungry for labeled data.
Today’s DL is supervised learning.
“The Revolution Will Not be Supervised.” –Yann Lecun
32
![Page 33: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/33.jpg)
Deep RL Takes Too Long to Train
RL systems require a gazillion trials!
33
![Page 34: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/34.jpg)
Neural Nets need a Vapnik!
The theory about generalization properties of ANNs is not completely understood.
34
![Page 35: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/35.jpg)
Neural Nets need a Vapnik! 35
![Page 36: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/36.jpg)
Natural Language Understanding
◻ Headlines: ⬜ Enraged Cow Injures Farmer With Ax
⬜ Hospitals Are Sued by 7 Foot Doctors
⬜ Ban on Nude Dancing on Governor’s Desk
⬜ Iraqi Head Seeks Arms
⬜ Local HS Dropouts Cut in Half
⬜ Juvenile Court to Try Shooting Defendant
⬜ Stolen Painting Found by Tree
⬜ Kids Make Nutritious Snacks
◻ Why are these funny?
Source: CS188
36
![Page 37: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/37.jpg)
Common Sense Knowledge
“There’ll be a lot of people who argue against it, who
say you can’t capture a thought like that. But there’s no
reason why not. I think you can capture a thought by a
vector.” – Geoff Hinton
"If a mother has a son, then the son is younger than the
mother and remains younger for his entire life."
"If President Trump is in Washington, then his left foot
is also in Washington,"
37
![Page 38: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/38.jpg)
THANKS!
Eduardo Bezerra ([email protected])
PPCIC – CEFET/RJ
Programa de Pós-Graduação em
Ciência da Computação
http://eic.cefet-rj.br/ppcic
![Page 39: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/39.jpg)
Backup slides
![Page 40: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/40.jpg)
Credits: Alec Radford.
40
![Page 41: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/41.jpg)
Transfer Learning
41
![Page 42: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/42.jpg)
ML Tribes
◻ There are several ML tribes!
⬜ Symbolists (rule-based systems)
⬜ Evolutionists (evolutionary computation, GAs)
⬜ Analogists (SVMs, k-NN, …)
⬜ Bayesians (Bayes update rule)
⬜ Connectionists (ANNs)
Pedro Domingos
42
![Page 43: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/43.jpg)
Difficult vs Easy
◻ […] hard problems are easy and the easy problems are
hard. The mental abilities of a four-year-old –
recognizing a face, lifting a pencil, walking across a
room, answering a question – in fact solve some of the
hardest engineering problems [...], it will be the stock
analysts and petrochemical engineers and parole board
members who are in danger of being replaced by
machines. The gardeners, receptionists, and cooks are
secure in their jobs for decades to come. Steven Pinker
43
![Page 44: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/44.jpg)
Artificial Neural Nets (ANNs) 44
![Page 45: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/45.jpg)
Artificial Neuron
45
A model inspired in the real one
(biological neuron).
45
![Page 46: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/46.jpg)
Artificial Neuron - input
46
46
![Page 47: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/47.jpg)
Artificial Neuron – parameters
47
47
![Page 48: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/48.jpg)
Artificial Neuron – pre-activation
48
48
![Page 49: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/49.jpg)
Artificial Neuron – activation function
49 49
"The composition of linear
transformations is also a
linear transformation"
Nonlinearities are necessary
so that the network can
learn complex
representations of the data.
49
![Page 50: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/50.jpg)
Artificial Neural Net
50 Feedforward Neural Network
50
![Page 51: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/51.jpg)
AI Winters 51
![Page 52: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/52.jpg)
Artificial Neuron
1943
McCulloch-Pitts model (boolean functions, cycles)
52
![Page 53: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/53.jpg)
Perceptron
1957
NY Times: “the embryo of an electronic
computer that [the Navy] expects will be
able to walk, talk, see, write, reproduce
itself and be conscious of its existence.
Frank Rosenblatt
53
![Page 54: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/54.jpg)
Perceptrons are Deprecated!
1969
“[…] by the mid-1960s there had been a great many
experiments with perceptrons, but no one had been
able to explain why they were able to recognize
certain kinds of patterns and not others.”
54
![Page 55: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/55.jpg)
The First AI Winter
1974
DARPA cuts AI funding
1973
The Lighthill report
55
![Page 56: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/56.jpg)
Moravek’s paradox
◻ "it is comparatively easy to make
computers exhibit […] intelligence
tests or playing checkers, and
difficult or impossible to give them
the skills of a one-year-old when it
comes to perception and mobility."
1988
56
![Page 57: Artificial Neural Networks: Recent Advances, Current ...ebezerra/wp-content/uploads/2018/02/Talk-20… · Artificial Neural Networks: ... is a cornerstone of deep neural nets. 10](https://reader035.fdocuments.us/reader035/viewer/2022081615/5fd87da75fca1a09f1324222/html5/thumbnails/57.jpg)
The Second AI Winter
1990s
57