Making a Robotic Dog See and Hear
description
Transcript of Making a Robotic Dog See and Hear
![Page 1: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/1.jpg)
Making a Robotic DogSee and Hear
Daniel D. Lee
World of Science 2000
![Page 2: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/2.jpg)
Face recognition
Original image
Alternative images
![Page 3: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/3.jpg)
Terminator
Arnold is looking for you...
![Page 4: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/4.jpg)
Robots
Hollywood versus reality
Data
Gort
HAL
![Page 5: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/5.jpg)
Deep Blue
Computer beats world champion Gary Kasparov
![Page 6: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/6.jpg)
Complexity
Tic Tac Toe easy to program using brute force Deep Blue evaluated 200 million chess positions
per second
Tic Tac Toe
1
0
1
0
1
1
0
0
1
Number ofconfigurations
1968339
![Page 7: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/7.jpg)
Images
0 0
05
0 7
10
08
0 2
0 0
.
.
.
.
.
.
.
Pixel vector
Vector representation of pixel values(white=0.0, black=1.0).
![Page 8: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/8.jpg)
Combinatorial explosion
Impossible for a computer to search all possible images
2 pixels
422
3 pixels
823
images
images120400 1032
400 pixels
images
Age of universe: 1710 seconds
![Page 9: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/9.jpg)
The brain
Vision occupies a large fraction of our brains
![Page 10: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/10.jpg)
Neurons
Approximately 1012 neurons in a human brain
![Page 11: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/11.jpg)
Neuronal properties
Neurons communicate with each other using action potentials
![Page 12: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/12.jpg)
Circuit diagram
Complex and hierarchical organization.
(Felleman & Van Essen, 1991)
![Page 13: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/13.jpg)
Artificial neuron
Unit sums inputs x with synaptic weights w Nonlinear transformation
x1
Squashing function
w1
x2
x3
x4
x5
w5
Inputactivities
Synapticweights
Output
![Page 14: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/14.jpg)
Artificial neural network
Output layer
W11
WNM
Weights
tx
,
Transformation of input into output. Change synaptic weights to maximize performance.
Labelled data:
Input layer
Hidden layer
x2
x3
xN
t1
t2
x4
x1
Input Output
![Page 15: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/15.jpg)
Learning
How to set the connections between neurons to have the network do the right thing?
Output layer
W11
WNM
Weights
Input layer
Hidden layer
x2
x3
xN
t1
t2
x4
x1
![Page 16: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/16.jpg)
Optimization
Like climbing a mountain blindfolded. Small steps until top is reached.
Mount Everest Gradient ascent
![Page 17: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/17.jpg)
Robotic dog
Doesn’t have a name yet… any suggestions?
![Page 18: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/18.jpg)
Artificial sensorimotor system
Total cost of parts ~ $700 You too can build your own!
![Page 19: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/19.jpg)
Video tracking
![Page 20: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/20.jpg)
Video processing
Conversion of video images into luminance, color, and motion channels.
![Page 21: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/21.jpg)
Face recognition neural network
Learns to associate saliency with face.
![Page 22: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/22.jpg)
Unsupervised learning
Database containing many different faces.
![Page 23: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/23.jpg)
Learning parts of faces
![Page 24: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/24.jpg)
Parts representation
=
Computer automatically decomposes the images into their constituent parts.
W: 49 hidden units
V X
Original:
![Page 25: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/25.jpg)
Eye movements
Fast eye movements to scan visual environment
(Yarbus, 1967)
Eye muscles
![Page 26: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/26.jpg)
Goldfish eye movements
![Page 27: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/27.jpg)
Control of eye position
![Page 28: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/28.jpg)
Neural integrator
(Pastor, et al., 1994)
![Page 29: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/29.jpg)
Vestibular system
Sense of balance and seasickness
![Page 30: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/30.jpg)
Vestibular-ocular reflex
![Page 31: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/31.jpg)
Auditory localization
(Konishi, 1990)Barn Owl
![Page 32: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/32.jpg)
Auditory localization
![Page 33: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/33.jpg)
Walking
![Page 34: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/34.jpg)
Language
dogs
jumped
lazy
Text Corpus
brown fox
Text Document
Model text document as collections of words.
Doc #1Doc #2
Doc #3
Doc #4 Doc #5
![Page 35: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/35.jpg)
Text and images analogy
X
1 0 0
0 2 1
1 0 1
Word counts:
Documents
Wor
ds
Text Images
words
document
wordfrequency
pixels
picture
grayscaleintensity
Represent documents with word frequencies. Analogy between learning algorithms.
![Page 36: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/36.jpg)
Learned semantic topics
courtgovernmentcouncilculturesupremeconstitutionalrightsjustice
presidentservedgovernorsecretarysenatecongresspresidentialelected
flowersleavesplantperennialflowerplantsgrowingannual
diseasebehaviorglandscontactsymptomsskinpaininfection
president (148)congress (124)power (120)united (104)constitution (81)amendment (71)government (57)law (49)
Entry on “Constitutionof the United States”
Grolier encyclopedia: 15276 words, 30991 articles. Semantic features, word sense disambiguation.
metal process method paper … glass copper lead steel
person example time people … rules lead leads law
![Page 37: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/37.jpg)
Multimodal integration
(Knudson, 1997)
Vision, hearing and language combined
![Page 38: Making a Robotic Dog See and Hear](https://reader035.fdocuments.us/reader035/viewer/2022062519/56814cff550346895dba283d/html5/thumbnails/38.jpg)
Summary
Adaptation and learning in biological systems important for vision, hearing, motor control.
Mimic neural systems in computer algorithms. Robotic systems can learn from experience. But still cannot compete with your family dog or
cat...